Systems, methods, apparatuses, and computer-readable media for image management in image-guided medical procedures

ABSTRACT

Presented herein are methods, systems, devices, and computer-readable media for image management in image-guided medical procedures. Some embodiments herein allow a physician to use multiple instruments for a surgery and simultaneously provide image-guidance data for those instruments. Various embodiments disclosed herein provide information to physicians about procedures they are performing, the devices (such as ablation needles, ultrasound transducers or probes, scalpels, cauterizers, etc.) they are using during the procedure, the relative emplacements or poses of these devices, prediction information for those devices, and other information. Some embodiments provide useful information about 3D data sets and allow the operator to control the presentation of regions of interest. Additionally, some embodiments provide for quick calibration of surgical instruments or attachments for surgical instruments.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.13/014,587, filed Jan. 26, 2011, which is a continuation-in-part of U.S.patent application Ser. No. 12/703,118 claims, filed Feb. 9, 2010,entitled Systems, Methods, Apparatuses, and Computer-Readable Media forImage Guided Surgery, which in turn claims priority benefit to U.S.Provisional Application No. 61/207,593, filed Feb. 17, 2009, U.S.Provisional Application No. 61/207,589, filed Feb. 17, 2009, and U.S.Provisional Application No. 61/207,592, filed Feb. 17, 2009, U.S. patentapplication Ser. No. 13/014,587 also claims priority benefit to U.S.Provisional Application No. 61/322,991, filed Apr. 12, 2010, and U.S.Provisional Application No. 61/387,132, filed Sep. 28, 2010. Allaforementioned provisional and non-provisional patent applications areincorporated by reference herein in their entirety for all purposes.

FIELD

The embodiments disclosed relate to computer-assisted surgery and morespecifically related to systems, methods, apparatuses, andcomputer-readable media for image management in image-guided medicalprocedures.

BACKGROUND

The past few decades have seen incredible development of technology andsystems for computer assisted, image based, or image-guided surgery. Theadvances in image-guided surgery are tied in part to technological andscientific improvements in imaging and 3D computer graphics. Forexample, the early work of Mark Levoy, Turner Whitted, Richard Holloway,and Stephen Pizer in the late 1980s provided new 3D computer graphicsrendering techniques, medical image shape detection, and head-mounteddisplays. These are some of the building blocks of later image-guidedsurgery systems built at the University of North Carolina in the mid1990s and after.

Image-guided surgery makes use of imaging to aid the surgeon to performmore effective or more accurate surgery. As merely one example of suchimage-guided surgery, the use of ultrasound to guide needles beinginserted into the liver for ablation are used by the surgeon to helpguide the needle.

Current image-guided surgery systems, however, have inadequate imagemanagement. For example, many systems require a surgeon or otherpractitioner to forego performing other activities during a procedure inorder to manipulate, using a mouse-and-monitor interface, the view of CTscans, MRI images, and other visualizable data. This suspension ordisruption of the procedure can take time and reduce fluidity. Anotherissue with some systems is that they do not provide sufficient controland flexibility in viewing CT scans and other visualizable medical datadue to unintuitive and difficult interfaces. Yet another problem withcurrent 3D viewing systems is that they display visualizable medicaldata in a way that obscures those areas in which the operator isinterested.

One or more of these problems and others are addressed by the systems,methods, devices computer-readable media, and other embodimentsdescribed herein. That is, some of the embodiments may address one ormore issue, while other embodiments may address different issues.

SUMMARY

Presented herein are methods, systems, devices, and computer-readablemedia for image management in image-guided medical procedures. In someembodiments, pose information for a set of 3D visualizable medical dataare determined along with real-time pose information for a medicaldevice. A region of interest for the set of 3D visualizable medical datamay be determined based on the real-time pose information for themedical device and the pose information for the set of 3D visualizablemedical data. Image guidance information may be generated based at leaston the 3D visualizable medical data in the region of interest. Agraphical rendering of the image guidance information may be displayedon one or more displays.

In some embodiments, a system may determine device type information fora first medical device; real-time emplacement information for the firstmedical device; and real-time emplacement information for a secondmedical device. The system may also determine the real-time relativeemplacements of the first and second medical devices with the computersystem and real-time prediction information for the first medicaldevice. The image guidance system may then generate image guidanceinformation based on the real-time relative emplacements of the firstand second medical devices, the real-time prediction information for thefirst medical device, and data related to the second medical device. Agraphical rendering of the image guidance information may be displayedon one or more displays.

Numerous other embodiments are described throughout herein. Althoughvarious embodiments are described herein, it is to be understood thatnot necessarily all objects, advantages, features or concepts need to beachieved in accordance with any particular embodiment. Thus, forexample, those skilled in the art will recognize that the invention maybe embodied or carried out in a manner that achieves or optimizes oneadvantage or group of advantages as taught or suggested herein withoutnecessarily achieving other objects or advantages as may be taught orsuggested herein.

All of these embodiments are intended to be within the scope of theinvention herein disclosed. These and other embodiments will becomereadily apparent to those skilled in the art from the following detaileddescription having reference to the attached figures, the invention notbeing limited to any particular disclosed embodiment(s).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the display of 3D volumetric data as planar dataalong three axes.

FIG. 2 illustrates the display of 3D volumetric data as a 3D block ofdata.

FIG. 3A illustrates a first exemplary system for image-guided medicalprocedures.

FIG. 3B illustrates a second exemplary system for image-guided medicalprocedures.

FIG. 4 illustrates a first example of displaying image guidance data.

FIG. 5 illustrates a second example of displaying image guidance data.

FIG. 6 illustrates a third example of displaying image guidance data.

FIG. 7 illustrates a fourth example of displaying image guidance data.

FIG. 8 illustrates a fifth example of displaying image guidance data.

FIG. 9 illustrates a sixth example of displaying image guidance data.

FIG. 10 illustrates a seventh example of displaying image guidance data.

FIG. 11 illustrates an eighth example of displaying image guidance data.

FIG. 12 illustrates a ninth example of displaying image guidance data.

FIG. 13 illustrates a tenth example of displaying image guidance data.

FIG. 14 illustrates an eleventh example of displaying image guidancedata.

FIG. 15 illustrates a twelfth example of displaying image guidance data.

FIG. 16 illustrates a thirteenth example of displaying image guidancedata.

FIG. 17 illustrates a fourteenth example of displaying image guidancedata.

FIG. 18 illustrates a fifteenth example of displaying image guidancedata.

FIG. 19 illustrates a sixteenth example of displaying image guidancedata.

FIG. 20 illustrates a first calibration device for image-guided medicalprocedures.

FIG. 21 illustrates a second calibration device for image-guided medicalprocedures.

FIG. 22 illustrates an example of calibrating devices for image-guidedmedical procedures.

FIG. 23 illustrates an example of method for image management inimage-guided medical procedures.

FIG. 24 illustrates a seventeenth example of displaying image guidancedata.

FIG. 25 illustrates an eighteenth example of displaying image guidancedata.

FIG. 26 illustrates a nineteenth example of displaying image guidancedata.

FIG. 27 illustrates a twentieth example of displaying image guidancedata.

FIG. 28 illustrates a twenty-first example of displaying image guidancedata.

FIG. 29 illustrates a twenty-second example of displaying image guidancedata.

FIG. 30 illustrates a twenty-third example of displaying image guidancedata.

FIG. 31 illustrates a twenty-fourth example of displaying image guidancedata.

FIG. 32 illustrates a twenty-fifth example of displaying image guidancedata.

FIG. 33 illustrates a twenty-sixth example of displaying image guidancedata.

FIG. 34 illustrates a twenty-seventh example of displaying imageguidance data.

FIG. 35A illustrates a twenty-eighth example of displaying imageguidance data.

FIG. 35B illustrates a twenty-ninth example of displaying image guidancedata.

FIGS. 36A-36D illustrates four additional examples of displaying imageguidance data.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS Overview

The various embodiments herein may allow an operator, surgeon, or othermedical practitioner to manipulate a surgical or medical device in orderto vary the display of a region of interest of 3D visualizable medicaldata. For example, FIG. 24 depicts a needle 2445 being displayed on adisplay 2420. The displayed needle 2445 may correspond to a trackedphysical needle in the operating room or other surgical space. Theoperator may be able to manipulate the real needle in order to move thevirtual needle 2445 displayed on display 2420. The needle may define aregion of interest for displaying a set of 3D visualizable medical data,such as a CT scan or MRI.

In an embodiment depicted in FIG. 24, a single planar or nearly planarimage or volume 2460 of the 3D visualizable medical data is displayed ina plane defined by needle 2445. As an operator manipulates needle 2445,the planar or nearly planar image 2460 may move and rotate with theneedle. Therefore, if the needle 2445 was rotated clockwise, the planarimage 2460 of the 3D visualizable medical data would also be rotated. Inrotating 2460, a new image or a new “slice” of the volume of 3Dvisualizable medical data would be seen. That is, in some embodiments,the image 2460 depicts a controllable “window into” or “slice of” the 3Dvisualizable medical data. This is discussed more below. In theembodiment shown in FIG. 24, there is also a second set visualizablemedical data displayed in region of interest 2462. The second image 2462may also be manipulatable, for example, by manipulating an ultrasoundtransducer (not depicted in FIG. 24). In allowing an operator tomanipulate both an ultrasound transducer and a needle 2445, the operatormay be able to view the correspondence between the visualizable medicaldata in region of interest 2462 and in region of interest 2460.

In some embodiments, just the region of interest 2460 of 3D visualizablemedical data may be shown and needle 2445 may be omitted. Further, insome embodiments, there is only one region of interest 2460 shown, andthe visualizable medical data 2462 is not depicted. Such embodimentswill allow an operator to intuitively “move through” a 3D set ofvisualizable medical data. In various embodiments, visualizable medicaldata 2462 may be the output of an ultrasound transducer. In otherembodiments, the visualizable medical data 2462 may be a region ofinterest in a second set of 3D visualizable medical data. For example,2460 could be the visualization of a preoperative CT scan while 2462could be the visualization of a single plane of image data from a 3Dultrasound transducer.

Whether or not one or two regions of interest are shown and whether ornot medical devices are also displayed, the system may allow an operatorto selectively move and display 3D visualizable data set in an intuitiveway. For example, an operator, such as a radiologist, may manipulate amedical device in order to move the region of interest inside a CT scanor MRI image. In doing so, the operator may be able to better view anddetect anatomical structures such as organs or blood vessels as well asdetect tumors, cysts, and other objects of interest. The intuitivehand-eye interface will also allow an operator to more easily build up amental model of what is seen inside.

Example Procedures

As another example, consider the standard of care for hepatic tumorablation: Before the procedure, a patient might undergo a CT or MRI scan(a first set of 3D visualizable medical data); and a radiologist maystudy the image and annotate it with locations of possible tumors(either as part of the first set of 3D visualizable medical data or as aseparate, second set of 3D visualizable medical data—other techniquesmay also be used for annotation, such as those discussed in ImageAnnotation in Image-Guided Medical Procedures, to Sharif Razzaque etal., Attorney Docket Number INOPT.032A, filed concurrently herewith). Attreatment time—whether percutaneous or surgical—the physician may find,examine, biopsy, and treat these tumors. Between the time of the CT scan(and annotations) and the time of the procedure, the anatomy may change(e.g. tumors grow, organs change shape and position, etc.). During theprocedure, the physician actively manipulates the organs, furtherchanging their shapes and positions. Because of these changes,ultrasound imaging may be used for guidance during the procedure, andmay permit the physician to visualize internal anatomy in real time intheir locations at the time of the procedure.

Some embodiments herein track the position and orientation of theultrasound probe and display the ultrasound scan oriented in 3D relativeto the physician. This spatial coherence makes it easier for thephysician to interpret the ultrasound scan. Moreover, the “fused” orjoint display of the ultrasound and CT (which illustrates the tumor andsurrounding anatomy) may allow the physician to (a) know in whatdirection to move the ultrasound transducer to find the tumor; (b)determine whether a lesion spotted in ultrasound is a tumor targeted inthe pre-op CT; and/or (c) avoid critical vessels and other structureswhile guiding the needle to the tumor.

Some embodiments' volumetric visualizations (see, e.g., FIGS. 23-36D andrelated text) allow the physician to focus only on the critical elementsof the procedure. Some embodiments show a stereoscopic 3D volumerendering of the CT image with its annotations, spatially registered tothe ultrasound scan. Traditionally, in order to keep specific organsfrom visually occluding the ones behind them, the clinician must eithersegment the organs/structures or specify an opacity map. Both operationsare tedious and time-consuming processes that often make volumerendering impractical for everyday clinical use. Some embodiments avoidthese problems by rendering opaquely and in sharp detail only thoseportions of the CT image located in the vicinity of the ultrasound probeand/or the needle's trajectory. The remaining portions of the CT imageare rendered increasingly transparently and out-of-focus the furtherthey are from the area of interest, so that they do not distract orocclude other structures, while still providing spatial context. Someembodiments make the physician aware of CT image features andannotations located in front of or behind the plane of the ultrasoundscan. If she chooses to inspect these in more detail, she simply movesthe ultrasound probe towards them. Given that during surgery aphysician's attention is mostly focused on the tissue near theultrasound probe, she directly benefits from visualization withoutseparately altering data visualization.

In particular, some embodiments aid doctors in the common situationwhere there are small tumors that are difficult to localize usingultrasound. With immediate guidance from the annotated CT in correctspatial arrangement with the ultrasound, the surgeon may know in whichdirection to move the ultrasound transducer in order to find the tumor.When a patient has many small tumors, embodiments also help the surgeondetermine correspondence—e.g., determining which of the 14 tumors in theCT image is the one currently visible in the ultrasound scan. In thisway, the physician may be able to ablate the tumors more thoroughly,confidently and rapidly, reducing the chance of recurrence.

The user, by observing previously-made annotations, could determine thecenter point of the tumor or organ, and mark that center point in the 3Dimage, so that she can guide the ablation needle to that center point.Furthermore, some embodiments compute the volume of tissue enclosed bythe 3D outline specified by the annotations, and compare that to thevolume that would be ablated by the ablation device. Displaying thevolume of tumor and the volume of healthy tissue enclosed in theablation zone may allow the user to reposition the ablation needle tomaximize the tumor volume ablation and minimize the healthy tissueablated.

Exemplary Systems

FIG. 3A illustrates a first exemplary system for image management inimage-guided medical procedures. FIG. 3B illustrates a second exemplarysystem for image management in image-guided medical procedures. In manyrespects the embodiments illustrated by FIGS. 3A and 3B are similar anduse similar numbering. Where the two are different, those differencesare noted. The differences between the two figures may include that, inFIG. 3A, two position sensing units 310 and 340 are shown, whereas inFIG. 3B, only a single position sensing unit 310 is shown.

In some embodiments, position sensing units 310 and 340 may be trackingsystems 310 and 340 and may track surgical instruments 345 and 355 andprovide data to the image guidance unit 330. The image guidance unit 330may process or combine the data and show image guidance data on display320. This image guidance data may be used by a physician to guide aprocedure and improve care. There are numerous other possibleembodiments of system 300. For example, many of the depicted modules maybe joined together to form a single module and may even be implementedin a single computer or machine. Further, position sensing units 310 and340 may be combined and track all relevant surgical instruments 345 and355, as discussed in more detail below and exemplified in FIG. 3B.Additional imaging units 350 may be included and combined imaging datafrom the multiple imaging units 350 may be processed by image guidanceunit 330 and shown on display unit 320. Additionally, two or moresurgical systems 349 may also be included.

Information about and from multiple surgical systems 349 and attachedsurgical instruments 345 may be processed by image guidance unit 330 andshown on display 320. These and other possible embodiments are discussedin more detail below. Imaging unit 350 may be coupled to image guidanceunit 330. In some embodiments, imaging unit 350 may be coupled to asecond display unit 351. The second display unit 351 may display imagingdata from imaging unit 350. The imaging data displayed on display unit320 and displayed on second display unit 351 may be, but are notnecessarily, the same. In some embodiments, the imaging unit 350 is anultrasound machine 350, the movable imaging device 355 is an ultrasoundtransducer 355 or ultrasound probe 355, and the second display unit 351is a display associated with the ultrasound machine 350 that displaysthe ultrasound images from the ultrasound machine 350. In someembodiments, a movable imaging unit 355 may not be connected directly toan imagining unit 350, but may instead be connected to image guidanceunit 330. The movable imaging unit 355 may be useful for allowing a userto indicate what portions of a first set of imaging data should bedisplayed. For example, the movable imaging unit 355 may be anultrasound transducer 355 or a tracked operative needle or other device355, for example, and may be used by a user to indicate what portions ofimaging data, such as a pre-operative CT scan, to show on a display unit320 as image 325. Further, in some embodiments, there could be a thirdset of pre-operative imaging data that could be displayed with the firstset of imaging data.

In some embodiments, system 300 comprises a first position sensing unit310, a display unit 320, and second position sensing unit 340 (if it isincluded) all coupled to image guidance unit 330. In some embodiments,first position sensing unit 310, display unit 320, and image guidanceunit 330 are all physically connected to stand 370. Image guidance unit330 may be used to produce images 325 that are displayed on display unit320. The images 325 produced on display unit 320 by the image guidanceunit 330 may be determined based on ultrasound or other visual imagesfrom the first surgical instrument 345 and second surgical instrument355. For example, if the first surgical instrument 345 is an ablationneedle 345 and the second surgical instrument 355 is an ultrasound probe355, then images 325 produced on display 320 may include the video fromthe ultrasound probe 355 combined with graphics, such as projectedneedle drive or projected ablation volume, determined based on theemplacement of ablation needle 345. If the first surgical instrument 345is an ultrasound probe 345 and the second surgical instrument 355 is alaparoscopic camera 355, then images 325 produced on display 320 mayinclude the video from the laparoscopic camera 355 combined withultrasound data superimposed on the laparoscopic image. More surgicalinstruments may be added to the system. For example, the system mayinclude an ultrasound probe, ablation needle, laparoscopic camera,cauterizer, scalpel and/or any other surgical instrument or medicaldevice. The system may also process and/or display collected data, suchas preoperative CT scans, X-Rays, MRIs, laser scanned 3D surfaces etc.

The term “emplacement” and the term “pose” as used herein are broadterms encompassing their plain and ordinary meanings and may refer to,without limitation, position, orientation, the combination of positionand orientation, or any other appropriate location information. In someembodiments, the imaging data obtained from one or both of surgicalinstruments 345 and 355 may include other modalities such as a CT scan,MRI, open-magnet MRI, optical coherence tomography, positron emissiontomography (“PET”) scans, fluoroscopy, ultrasound, or otherpreoperative, or intraoperative 2D or 3D anatomical imaging data. Insome embodiments, surgical instruments 345 and 355 may also be scalpels,implantable hardware, or any other device used in surgery. Anyappropriate surgical system 349 or imaging unit 350 may be attached tothe corresponding medical instruments 345 and 355.

As noted above, images 325 produced may also be generated based on live,intraoperative, or real-time data obtained using second surgicalinstrument 355, which is coupled to second imaging unit 350. The term“real time” as used herein is a broad term and has its ordinary andcustomary meaning, including without limitation instantaneously ornearly instantaneously. The use of the term real time may also mean thatactions are performed or data is obtained with the intention to be usedimmediately, upon the next cycle of a system or control loop, or anyother appropriate meaning. Additionally, as used herein, real-time datamay be data that is obtained at a frequency that would allow a surgeonto meaningfully interact with the data during surgery. For example, insome embodiments, real-time data may be a medical image of a patientthat is updated one time per second. In some embodiments, real-time datamay be ultrasound data that is updated multiple times per second.

Second surgical instrument 355 may be coupled to second position sensingunit 340. Second position sensing unit 340 may be part of imaging unit350 or it may be separate. Second position sensing unit 340 may be usedto determine the emplacement of second surgical instrument 355. In someembodiments, first and/or second position sensing units 310 and/or 340may be magnetic trackers and magnetic may be coils coupled to surgicalinstruments 345 and/or 355. In some embodiments, first and/or secondposition sensing units 310 and/or 340 may be optical trackers andvisually-detectable fiducials may be coupled to surgical instruments 345and/or 355.

Images 325 may be produced based on intraoperative or real-time dataobtained using first surgical instrument 345, which is coupled to firstsurgical system 349. In FIGS. 3A and 3B, first surgical system 349 isshown as coupled to image guidance unit 330. The coupling between thefirst surgical system 349 and image guidance unit 330 may not be presentin all embodiments. In some embodiments, the coupling between firstsurgical system 349 and image guidance unit 330 may be included whereinformation about first surgical instrument 345 available to firstsurgical system 349 is useful for the processing performed by imageguidance unit 330. For example, in some embodiments, first surgicalinstrument 345 is an ablation needle 345 and first surgical system 349is an ablation system 349. In some embodiments, it may be useful to senda signal about the relative strength of planned ablation from ablationsystem 349 to image guidance unit 330 in order that image guidance unit330 can show a predicted ablation volume. In other embodiments, firstsurgical system 349 may not be coupled to image guidance unit 330.Example embodiments including images and graphics that may be displayedare included below.

In some embodiments, first position sensing unit 310 tracks theemplacement of first surgical device 345. First position sensing unit310 may be an optical tracker 310 and first surgical device 345 may haveoptical fiducials attached thereto. The emplacement of optical fiducialsmay be detected by first position sensing unit 310, and, therefrom, theemplacement of first surgical device 345 may be determined.

In various embodiments, as depicted in FIG. 3B, a single positionsensing unit 310 may track both first medical device 345 and secondmedical device 355. In FIG. 3B, in some embodiments, position sensingunit 310 is a magnetic tracker and is mounted below a surgical table380. Such an arrangement may be useful when the tracking volume of theposition sensing unit 310 is dependent on the location of the positionsensing unit, as with many magnetic trackers. Magnetic tracking coilsmay be mounted in or on the medical devices 345 and 355.

In some embodiments, either or both of the first position sensing unit310 and the second position sensing unit 340 may be an Ascension Flockof Birds, Nest of Birds, driveBAY, medSAFE, trakSTAR, miniBIRD,MotionSTAR, pciBIRD, or Calypso 4D Localization System and trackingunits attached to the first and/or second surgical or medical devices345 and 355 may be magnetic tracking coils. The term “tracking unit,” asused herein, is a broad term encompassing its plain and ordinary meaningand includes without limitation all types of magnetic coils or othermagnetic field sensing devices for use with magnetic trackers, fiducialsor other optically detectable markers for use with optical trackers,such as those discussed above and below. Tracking units could alsoinclude optical position sensing devices such as the HiBall trackingsystem and the first and second position sensing units 310 and 340 maybe part of a HiBall tracking systems. Tracking units may also include aGPS device or signal emitting device that would allow for tracking ofthe position and, optionally, orientation of the tracking unit. In someembodiments, a signal emitting device might include a radio-frequencyidentifier (RFID). In such embodiments, the first and/or second positionsensing unit 310 and 340 may take in the GPS coordinates of the trackingunits or may, for example, triangulate the radio frequency signal beingemitted by the RFID associated with tracking units. The tracking systemsmay also include one or more 3D mice.

In some embodiments, either or both of the first position sensing unit310 and the second position sensing unit 340 may be an Aurora®Electromagnetic Measurement System using sensor coils for tracking unitsattached to the first and/or second surgical devices 345 and 355. Insome embodiments, either or both of the first position sensing unit 310and the second position sensing unit 340 may also be an optical 3Dtracking system using fiducials. Such optical 3D tracking systems mayinclude the NDI Polaris Spectra, Vicra, Certus, PhaseSpace IMPULSE,Vicon MX, InterSense IS-900, NaturalPoint OptiTrack, Polhemus FastTrak,IsoTrak, or Claron MicronTracker2. In some embodiments, either or bothof position sensing units 310 and 340 may each be an inertial 3Dtracking system comprising a compass, accelerometer, tilt sensor and/orgyro, such as the InterSense InertiaCube or the Nintendo Wii controller.In some embodiments, either or both of position sensing units 310 and340 may be attached to or affixed on the corresponding surgical device345 and 355. In some embodiments, the position sensing units, 310 and340, may include sensing devices such as the HiBall tracking system, aGPS device, or signal emitting device that would allow for tracking ofthe position and, optionally, orientation of the tracking unit. In someembodiments, a position sensing unit 310 or 340 may be affixed to eitheror both of the surgical devices 345 and 355. The surgical devices 345 or355 may be tracked by the position sensing units 310 or 340. A roomcoordinate system reference, such as the display 320 may also be trackedby the position sensing unit 310 or 340 in order to determine theemplacements of the surgical devices 345 and 355 with respect to theroom coordinate system. Devices 345 and 355 may also include or havecoupled thereto one or more accelerometers, which may be used toestimate movement, position, and location of the devices.

In some embodiments, the display unit 320 displays 3D images to a user,such as a physician. Stereoscopic 3D displays separate the imagery shownto each of the user's eyes. This can be accomplished by a stereoscopicdisplay, a lenticular auto-stereoscopic display, or any otherappropriate type of display. The display 320 may be an alternating rowor alternating column display. Example alternating row displays includethe Miracube G240S, as well as Zalman Trimon Monitors. Alternatingcolumn displays include devices manufactured by Sharp, as well as many“auto-stereoscopic” displays (e.g., Philips). Display 320 may also be acathode ray tube. Cathode Ray Tube (CRT) based devices, may use temporalsequencing, showing imagery for the left and right eye in temporalsequential alternation; this method may also be used by newer,projection-based devices, as well as by 120-Hz-switchable liquid crystaldisplay (LCD) devices.

In some embodiments, a user may wear a head mounted display in order toreceive 3D images from the image guidance unit 330. In such embodiments,a separate display, such as the pictured display unit 320, may beomitted. The 3D graphics may be produced using underlying data models,stored in the image guidance unit 330 and projected onto one or more 2Dplanes in order to create left and right eye images for a head mount,lenticular, or other 3D display. The underlying 3D model may be updatedbased on the relative emplacements of the various devices 345 and 355,as determined by the position sensing unit(s), and/or based on new dataassociated with the devices 345 and 355. For example, if the seconddevice is an ultrasound probe 355, then the underlying data model may beupdated to reflect the most recent ultrasound image. If the first device345 is an ablation needle, then the underlying model may be updated toreflect any changes related to the needle, such as power or durationinformation. Any appropriate 3D graphics processing may be used forrendering including processing based on OpenGL, Direct3D, Java 3D, etc.Whole, partial, or modified 3D graphics packages may also be used, suchpackages including 3DS Max, SolidWorks, Maya, Form Z, Cybermotion 3D,VTK, Slicer, or any others. In some embodiments, various parts of theneeded rendering may occur on traditional or specialized graphicshardware. The rendering may also occur on the general CPU, onprogrammable hardware, on a separate processor, be distributed overmultiple processors, over multiple dedicated graphics cards, or usingany other appropriate combination of hardware or technique.

One or more modules, units, devices, or elements of various embodimentsmay be packaged and/or distributed as part of a kit. For example, in oneembodiment, an ablation needle, tracking elements, 3D viewing glasses,and/or a portion of an ultrasound wand may form a kit. Other embodimentsmay have different elements or combinations of elements grouped and/orpackaged together. Kits may be sold or distributed separately from orwith the other portions of the system.

There are numerous other examples of image guidance systems which mayuse, incorporate, support, or provide for the techniques, methods,processes, and systems described herein, such as the 3Dcomputer-graphics-based assigned to InnerOptic Technologies, Inc. thatprovides for displaying guidance data from multiple sources, U.S.application Ser. No. 11/833,134, filed Aug. 2, 2007, the contents ofwhich are incorporated by reference herein in their entirety for allpurposes. The image guidance may also be performed at least in partusing the techniques described in U.S. patent application Ser. No.11/828,826, filed Jul. 26, 2007, U.S. Pat. No. 7,728,868, U.S. patentapplication Ser. No. 12/399,899, U.S. patent application Ser. No.12/483,099, U.S. patent application Ser. No. 12/893,123, U.S. patentapplication Ser. No. 12/842,261, and/or U.S. patent application Ser. No.12/703,118, each of which is incorporated by reference herein in itsentirety for all purposes. Also incorporated by reference for allpurposes is Image Annotation in Image-Guided Medical Procedures, toRazzaque et al., filed concurrently herewith.

Methods for Image Management in Image-Guided Medical Procedures

FIG. 23 illustrates a method 2300 for image management in image-guidedmedical procedures. In block 2310, pose information for a set of 3Dvisualizable medical data is determined. The term “3D visualizablemedical data” is a broad that encompasses its ordinary and customarymeaning, and includes, without limitation any data in a volume or 3Dspace that can be displayed. The 3D visualizable medical data caninclude, without limitation, one or more of a CT scan, an MRI, other 3Dpreoperative imaging data, other volume data, segmented internal organs,segmented blood vessels, annotations, tumors, etc. Determining poseinformation for the set of 3D visualizable medical data in block 2310may be done only once initially in order to register or approximatelyregister the 3D visualizable medical data with the medical scene beingvisualized for the operator. Various techniques for registering the 3Dvisualizable medical data with the medical scene may be used includingmatching features in the 3D visualizable medical data with featuresknown to be in the medical scene at that time, such as tumors, bones,blood vessels, etc. Manual registration may also be possible where anoperator or other technician manipulates the pose of the 3D visualizablemedical data relative to the scene. After the pose for the 3Dvisualizable medical data is known, subsequent steps or blocks ofdetermining pose information for the 3D visualizable medical data maysimply be referring to the previously determined pose. This may be thecase, for example, if the 3D visualizable medical data does not moveover time. If the 3D visualizable medical data does move over time, thenthe pose for that 3D visualizable medical data may be determined aftereach such movement. The pose information may also be determined usingany tracking systems, devices, techniques, or methods, such as thosediscussed herein.

In block 2320, pose information is determined for a medical device. Asdiscussed elsewhere herein, “medical device” as used herein is a broadterm that encompasses, but is not limited to a device, item, or partused in a medical procedure. For example, a medical device could includean ablation needle, an ultrasound transducer, a cauterizer, a scalpel, aglove covering the operator's hand, the operator's hand or finger, etc.The medical device used for pose information could even be an operator'shead, eyes, or gaze direction. Pose information for the medical devicemay be obtained using any system, device, method, or technique, such asthose disclosed herein.

As depicted in FIG. 23, in at least one embodiment, pose information foradditional set(s) of 3D visualizable medical data may also bedetermined. This is depicted by the dashed arrows in FIG. 23 and block2311. Determining the pose information for the second or subsequent setsof 3D visualizable medical data in block 2311 may include the sametechniques described above with respect to block 2310. Further, the poseinformation for the additional sets of 3D visualizable medical data maybe registered with the medical scene and/or with the first 3Dvisualizable medical data. In this way, all the relative registration ormatrix transformation between the two (or more) 3D visualizable medicaldata sets will be known and the two sets of visualizable medical datawill be displayable relative to the operator.

In some embodiments, the first set of 3D visualizable medical data maybe a CT scan, MRI or other image from a first time period, and a secondor subsequent set of 3D visualizable medical data may be a CT scan, MRI,or other visualizable medical data from another later time period.Having these two or more sets of visualizable medical data may allow anoperator or other technician to view the changes over time in a person'sbody.

In one embodiment, as depicted in block 2321, real-time pose informationfor a second (third, fourth, etc.) medical device may also bedetermined. The second medical device, like the first medical device,could be any of many different objects in the medical scene includingdevices and portions of the operator or other technician, such as afinger, etc. The first and second medical devices may be tracked usingoptical tracking or magnetic tracking, by attaching radio emitters andperforming triangulation using a high ball or other tracking device, orusing any other tracking technique, such as those described or discussedherein.

After the pose information for visualizable medical data and real-timepose information for medical device(s) are known, then in block 2330 aregion of interest may be determined based on the pose information. Asdiscussed above, the region of interest may, for example, include themedical device and may define a volume or plane within the 3Dvisualizable medical data. For example, in FIG. 24, the region ofinterest 2460 includes the medical device 2445 and cuts through the 3Dvisualizable medical data for display on the screen 2420. The region ofinterest 2460 may be based on the relative poses of the 3D visualizablemedical data and the medical device 2445. For example, the region ofinterest may be defined by a plane passing through the axis of device2445. In some embodiments, the region of interest might be defined byboth device 2445 and a second device (not depicted). For example, if thesecond device is an ultrasound transducer associated with the region ofinterest 2462, the region of interest 2460 may be a plane defined by anaxis of the device 2445 and intersection with region of interest 2462.

Numerous other shapes and types of regions of interest are alsopossible. FIG. 24 shows a planar or nearly planar region of interest2460 defined by the pose of the medical device 2445. FIG. 25 shows dualregions of interest, one planar region of interest 2560 and one regionof interest 2564 that is a rectilinear volume that contains segmentedveins. Both of these may be defined by medical device 2545. FIG. 26depicts the volumetric region of interest 2670 defined by medical device2645 which is a rectilinear or cuboid shape. The segmented veins 2664inside of volume 2670 are those that are displayed to the operator. Aplanar CT scan's region of interest 2660 may also be displayed withinvolume 2670. FIG. 27 shows a cylindrical volumetric region of interest2770 defined by medical device 2745. Within that volume 2770, thesegmented veins 2764 are shown.

For some medical devices, in some embodiments, specific regions ofinterest may exist. For example, turning to FIG. 28, if medical device2845 is an ablation needle, then the region of interest 2870 may be theablation volume or the expected ablation volume of ablation needle 2845.This ablation volume may be displayed on display 2820. In FIG. 28, thereis also an ultrasound image being displayed as image 2862. It may beimportant for an operator to know what structures are within an ablationvolume 2870 in order to see what tissue is expected to be ablated. Forexample, an operator may want to avoid ablating a vein. Therefore,ablation volume 2870 may be useful in order to see whether any of thesegmented veins, such as those depicted in FIG. 27 as segmented veins2764, are within the ablation volume 2870 (in FIG. 28). In FIG. 28,there are no segmented veins displayed within the ablation volume 2870and therefore it may be clear to the surgeon or other operator to ablatetissue at that point. In one embodiment, the outlines shown aroundvolume 2870 and around the volumes in FIGS. 26 and 27 may or may not beshown. In another embodiment, as in FIG. 25, the volume is defined bymedical device 2545, but the outline of the volume is not displayed.

In an embodiment, a medical device may be an ultrasound probe and thevolume of the region of interest may be around the slice of ultrasounddata as depicted in FIG. 29 as volumetric region of interest 2970. Inthis case, the medical device 2945 is not defining the region ofinterest, but instead the undepicted ultrasound probe defines the regionof interest.

In some embodiments, dual or joint regions of interest will bedisplayed. For example, in FIG. 30 a first region of interest 3070 isdefined by an ultrasound probe that is not depicted in FIG. 30, and asecond region of interest 3071 is defined as an ablation volume ofablation needle 3045. Both of these regions of interest may be used tovisualize medical data from one or more sets of 3D visualizable medicaldata and displayed on screen 3020.

The region of interest may also be defined in part based on the screenplane or perpendicular to the view axis, that is, the direction in whichthe operator is viewing the medical scene. That is, for example, turningto FIG. 25, the medical device 2545 may define in part the region ofinterest 2560 and the region of interest 2560 may be perpendicular tothe view axis of the operator (not depicted in FIG. 25) or may beparallel to the screen 2520. In other embodiments, as discussed in partherein, the region of interest 2560 is not defined by the screen planeor the viewing axis and may move freely with the medical device 2545.

After the region of interest is determined based on the poses, imageguidance information is generated based on the region of interest inblock 2340. Generating the image guidance information based on theregion of interest can include determining, for example, an image thatrepresents the slice of the region of interest through the 3Dvisualizable medical data. Turning again to FIG. 24, the region ofinterest 2460 is a plane that has been cut through 3D visualizablemedical data. Determining that the appearance of that region of interest2460 can include determining the intensity, color, contrast, brightness,etc. of each voxel (or pixel, subpixel, or other unit) for that imagebased on the corresponding information in the 3D visualizable medicaldata. For example, if the 3D visualizable medical data is a CT scan,then determining the image guidance information in block 2340 mayinclude rendering a planar slice corresponding to the region of interestfrom the CT scan.

In FIG. 25, three different sets of visualizable medical data, 2560,2562 and 2564, are displayed on display 2520. The regions of interestmay be defined in part by medical device 2545, depicted as a needle. Thedevice 2545 may define a first region of interest 2560 through a CT scanand may define a second region of interest 2564 as a volume capturingthe segmented veins that are part of a larger volume of veins (notdepicted in FIG. 25). That is, there may be a large volume of 3Dvisualizable medical data containing numerous segmented veins or otherorgans and the region of interest defined by medical device 2545 maydefine which veins 2564 are displayed on display 2520.

In some embodiments, generating image guidance information based on theregion of interest may include generating image guidance data in whichone set of visualizable medical data or image may be displayed in theregion of interest and information outside the region of interest maydisplay a second set of 3D visualizable medical data. Consider, forexample, FIG. 36D, which displays an ultrasound probe 3655 which hasassociated with it an ultrasound image 3656. The image guidanceinformation generated is such that the ultrasound image 3656 isdisplayed, surrounded by 3D visualizable medical data 3670. The 3Dvisualizable medical data 3670 is displayed outside the region ofinterest 3660, thereby allowing an operator to see the contextsurrounding an ultrasound image. In such embodiments, the generatedimage guidance information may include the 3D visualizable medical data3670 that does not occlude or overlap with the rendering of the regionof interest 3660, thereby cutting out a “tunnel” in the 3D visualizablemedical data 3670 in order to view, without obstruction, the ultrasoundimage 3656. This can also be used even where multiple sets of 3Dvisualizable medical data 3670 are displayed together. A tunnel may becut through all of them in order to view the region of interest 3660.

Additional embodiments are also presented in FIGS. 36A-36C. FIG. 36Ashows image guidance information generated such that an ultrasound image3656 is displayed in region of interest 3660, and CT scan 3670 isdisplayed outside of the region of interest 3660. In FIGS. 36B and 36C,multiple planes or slices of CT data 3670 are generated as part of theimage guidance data. The planes or slices of CT data may be the originalslices of CT data, or they may be generated to be, e.g., parallel to theplane of the region of interest.

After the image guidance information has been generated based on theregion of interest in block 2340, then in block 2350 a graphicalrendering of the image guidance information is displayed. In someembodiments, the display of graphical information can be monoscopic orstereoscopic. Further, multiple rendering techniques may be used. Forexample, FIG. 31 depicts an example in which the edges or areas near theedges of the region of interest are displayed in a blurred or fadingmanner. That is, segmented veins 3164 as they reached the edge of theregion of interest defined by medical device 3145 are displayed asfading away on display 3120. Numerous other rendering techniques may beused. For example, FIG. 32 depicts an outlined depiction of anultrasound probe 3255 and a needle 3245, each of which are medicaldevices and each of which is associated with a region of interest. Inthis example, needle 3245 defines a region of interest 3260 andultrasound probe 3255 defines region of interest 3262. The data beingdisplayed may be part of a 3D ultrasound volume or it may be other datasuch as a CT scan, MRI or other 3D data set. The areas of overlap 3263and 3265 are displayed semi-transparently so that an operator maysimultaneously view both regions of interest (e.g., neither is obscuringthe other). Additional embodiments are depicted in FIG. 33 in whichultrasound probe 3355 is associated with region of interest 3362 andneedle 3345 is associated with region of interest 3360. Here again theoverlapped 3363 and 3365 are displayed semi-transparently so that theimage behind the overlapping portions of the image can also be seen.Non-overlapped regions (indicated by 3366 and 3367) are displayedopaquely, which, in some embodiments, will provide better image qualityin those regions 3366 and 3367.

In other embodiments, such as that depicted in FIG. 34, other renderingtechniques may be used. For example, ultrasound probe 3455 may beassociated with region of interest 3464 and that ablation needle 3445 isassociated with region of interest 3460. Here the areas of overlap 3463and 3465 are shown in a checkerboard fashion that may change over time.For example, in this embodiment, image 3460 is darker and that image3464 is lighter. In the overlapped portions 3463 and 3465, acheckerboard in which the image from behind is shown in one portion ofthe checkerboard and the image from the front is shown in the otherportion of the checkerboard. As this checkerboard moves over time, theoperator will be able to see, at different times, different portions ofboth images 3460 and 3464. These and other rendering techniques mayallow an operator to understand information in images that areoverlapping.

Other techniques for allowing or enabling an operator to viewoverlapping data on a screen are also possible. Considering, forexample, FIG. 35A in which textual information 3560, segmented veins3562, and a segmented organ 3564 are all shown in an overlapped fashion.In order to focus an operator's attention on the textual information3560, the textual information 3560 may be rendered in focus and theinformation behind. That is, vein 3562 and organ 3564 may be renderedout of focus. FIG. 35B depicts that, in order to focus the attention ofan operator on veins 3562, the veins 3562 are rendered in focus andtextual information 3560 and organ 3564 are rendered out of focus ondisplay 3520. It is also possible to enable the viewer to clearlydistinguish among these elements 3560, 3562 and 3564 by positioning themat different depths and using stereoscopic and/or head track displayqueues in order to have the user differentiate. Even if an object is notrendered in focus, the user can still choose to focus attention on aselected element by converging his or her eyes on it stereoscopicallyand/or by taking advantage of head motion parallax which inducesrelative motion between display elements located at different depths.This may allow an operator to single out a particular display element.Some embodiments that use an eye tracker may calculate the convergingpoint of the user's eyes through triangulation and adjust the focus anddepth of field accordingly. That is, if the user's eyes are converged atthe scanned depth of the textual information 3560 in FIG. 35A, then thattextual information 3560 may be rendered in focus and the otherinformation 3562 and 3564 may be rendered out of focus. If the user thenconverges his or her eyes on the segmented vein 3562, then as depictedin FIG. 35B, the vein 3562 may be rendered in focus and the textualinformation 3560 and organ 3564 may be rendered out of focus. In certainembodiments, it is also possible to manipulate a display's convergencefor easier stereoscopic fusion. Examples of this are in U.S. patentapplication Ser. No. 12/609,915 which published as U.S. PatentPublication No. 2010-0045783 on Feb. 25, 2010, which is herebyincorporated by reference for all purposes. In some embodiments, thefocus at the field and/or convergence are modulated or changedautomatically in order to direct the user's attention to specificdisplay elements. This may be accomplishable without any eye tracking.

FIG. 36D depicts yet another embodiment of displaying the generatedimage guidance information. FIG. 36 depicts an ultrasound probe 3655associated with an ultrasound image 3656. The ultrasound image 3656 isdisplayed notwithstanding the fact that it is within a region ofinterest 3660 showing 3D visualizable medical data, such as a CT scan.That is, the volume of data is displayed outside the region of interest3660, and the ultrasound image 3656 is displayed in the region ofinterest 3660. In some embodiments, this allows an operator to view anultrasound scan 3656 surrounded by, and in the context of, the 3Dvisualizable medical data. In some embodiments, the medical device maybe manipulated in order to show a tunnel to a region of interest even ifthe medical device is not associated with an image such as an ultrasoundimage. That is, the operator may want to simply “cut into” a 3D data setand provide a view on the inside of that 3D data set while still keepingthe context outside of the region of interest.

Turning to FIG. 36A, in some embodiments, a medical device may define aregion of interest 3660 in which an ultrasound image 3656 may be shownin focus. Outside the region of interest 3660, the CT scan 3670 may bedisplayed. As depicted in FIG. 36A, a single slice of CT data 3670 maybe displayed outside the region of interest 3660, or, as depicted inFIG. 36B, multiple slices of CT data 3670 may be displayed outside theregion of interest. Further, as depicted in FIG. 36C, the slices of CTdata may be rendered differently depending on the distance from theregion of interest. For example, planes of CT scan data 3670 may berendered more transparently (less brightly, etc) the further each isfrom the plane containing the region of interest. The slices of CT datamay be the slices from the underlying CT data, or the slices may begenerated to be, e.g., parallel or nearly parallel, with a planeassociated with the region of interest 3660. FIG. 36C also depicts thata tunnel may be cut through the rendered slices of the CT scan 3670 inorder to display the region of interest 3660 without or with littleoverlap. This tunnel may be altered as the region of interest or CT scandata are moved to always allow the operator to view the region ofinterest. FIG. 36D depicts a semi-realistic rendering of a CT scan 3670around a region of interest 3660. Inside the region of interest 3660, anultrasound image 3656 is displayed. Also displayed on display 3620 inFIG. 36D, is an outline of the medical device 3655.

The data shown in the region of interest may be any appropriatevisualizable medical data, not limited to ultrasound or CT data.Further, the data displayed outside of the region of interest may be anyvisualizable medical data, and may even be from the same data set as thedata shown in the region of interest. For example, MRI data may be shownin fading planes outside of the region of interest and in focus (andvisualizable through a tunnel) inside the region of interest. Further,annotation may be displayed along with the rendering of the visualizablemedical data inside and/or outside of the region of interest. In thismanner, an operator may see the annotations in the context of thevisualizable medical data.

In rendering the image guidance data, each point, line segment, splinesegment, point cloud, etc. may be made transparent and/or blurry basedon its distance from the region of interest, and its rendering may becontrolled using various graphic techniques, such as bit maps and pixelshaders, such as those discussed in Image Annotation in Image-GuidedMedical Procedures, to Sharif Razzaque et al., filed concurrentlyherewith, which is incorporated by reference above for all purposes.

In various embodiments, more than one set of visualizable medical datais rendered. Each one may be rendered in a different manner. Forexample, they may be rendered with different transparencies,brightnesses, contrast, colors, etc. Further, one or the other may berendered with a different transparency, brightness, contrast or color asdistance from the region of interest increases. For example, brightnessmay decrease and/or transparency may increase further from the region ofinterest

In an embodiment, displaying the graphical rendering of the imageguidance information comprises displaying current image guidanceinformation and gradually fading but still displaying previous imageguidance information. In this way, previous image guidance informationmay still appear on the screen but will fade over time and give anindication to a user or operator that the information is not as timely.

In some embodiments, annotations are received from an operator and theseannotations may be placed in 3D space. These annotations may also bedisplayed as part of block 2350 or may be displayed as part of anotherblock or in a different way. Further, as discussed above, annotationsmay be part of a 3D visualizable data set.

The blocks of process 2300 may be performed in a different order, may beaugmented by other blocks or may have sub-blocks within the blocksshown. Further, the process 2300 may be performed on a single computeror processor, on multiple computers or processors, on a single ormultiple virtual machines, and/or in a distributed fashion on multipleprocessors, machines, or virtual machines.

Limiting Data Export

In some embodiments, the 3D visualizable medical data will be from a 3Dultrasound scanner or other similar device that produces real-time 3Dimagery. Consider, for example, FIG. 33. A 3D ultrasound scanner may beable to image a volume 3370. Often 3D ultrasound scanners are limited bythe time required to export the 3D volume of visualizable data 3370 to acomputing system or other device. It is often not possible to export the3D volume of data 3370 continuously, in real-time, as it is imaged.Therefore, the volume 3370 might be exported from the ultrasound scanneroffline (after the procedure). In some embodiments herein, informationdefining one or more regions of interest, for example, 3362 and 3360 issent to the ultrasound scanner so that the ultrasound scanner might onlyexport data in those two regions of interest in real time. Therefore, anoperator may be able to receive from the 3D ultrasound scanner the tworegions of interest in real time without requiring the 3D ultrasoundexport the entire volume 3370.

Image Guidance Processes and Data

Overview

In some embodiments, a system, such as depicted in FIGS. 3A and 3B, maytrack a surgical instrument, such as an ablation needle, and an imagingdevice, such as an ultrasound probe. In other embodiments, numerousother medical or surgical devices may be tracked, and the system maytrack three, four, or more of medical or surgical devices or medicaldevices. A “medical device” is a term that may include surgical devicesand non surgical devices and all of those terms are broad terms and areintended to encompass their plain and ordinary meaning, includingwithout limitation, ablation needles, scalpels, endoscopes, ultrasoundprobes, etc. Information about the surgical or medical devices may bepredicted or determined and displayed by the system in 2D or 3D on adisplay visible to physician using the system. “Prediction information”is a broad term and is intended to encompass its plain and ordinarymeaning, including without limitation, image guidance data, mathematicaldata, mathematical or 3D models related to a procedure, etc.

As an example embodiment, the system may determine or predict theintersection point of an ultrasound slice and the projection of anablation needle. The system may also compute and display relativeorientation information, and, in the case of collected 3D data, such as3D ultrasound data, the system may segment and display the data indifferent manners, such as in 2D slices, 3D volumes, etc. This “extra”data displayed to the physician may be useful as it provides thephysician with more information about the operation, the instruments,and/or data from the instruments.

Embodiments herein may be used for many kinds of needle-based medicalprocedures, including, but not limited to, radiofrequency-, cryo-,microwave-, laser-ablation of tumors, fibroids, lesions, etc, as well asbiopsies, injections, central line placements, cyst aspirations, fluiddrainings, lumpectomies, and guidance of wires and stents through bloodvessels and ducts. Herein, the term needle to refer to any rigidneedle-like object, such as an ablation antenna or probe, cannula,catheter, electro-cautery device, Bovie, laser waveguide, stentapplication device, etc. Needle may also refer to a non-rigid or nearlyrigid version of the above. The system may also be used with non-needledevices such as scalpels, forceps, cutting loops on hysteroscopes,harmonic sheers, lasers (including CO₂ lasers), etc.

Some embodiments include tracking fixtures that may mount to surgical ormedical devices, such as ablation needles, ultrasound probes, ultrasoundprobes, scalpels, etc. that allow for quicker attachment and easiertracking calibration for the devices. These embodiments may allow aphysician to more quickly and easily start using the system.

Depicting Surgical Instruments

Previous systems do not provide satisfactory image guidance data. It canoften be difficult to discern the content of a 3D scene from a 2Ddepiction of it, or even from a 3D depiction of it. Therefore, variousembodiments herein provide image guidance that can help the doctorbetter understand the scene, relative emplacements or poses of object inthe scene and thereby provide improved image guidance. FIG. 4illustrates an image 401 of an exemplary surgical instrument 445 beingdisplayed on a screen 420. In this case, the surgical instrumentdisplayed is an ablation needle 445. Also depicted is the wire 446connecting the ablation needle to an ablation system. Some models ofneedles have markings such as bands around the shaft (to indicatedistance along the shaft), and a colored region near the tip to indicatewhere the radio frequency or microwave energy is emitted from in thecase of an ablation probe. Physicians performing needle procedures areoften familiar with these markings and may use them to help understandthe spatial relationship between the needle and anatomy. In someembodiments, the make and model of the needle 445 is known to the imageguidance system and the needle displayed (401) in display 420 mayresemble needle 445. The features of needles that may be rendered in thescene include the overall shape (diameter, cross sectional shape,curvature, etc.), color, distance markers, visuals or echogenicfiduciary markers, the state of deployable elements such as tines,paddles, anchors, resection loops, stiffening or steerable sleeves,temperature, radiation, light or magnetic field sensors, lens,waveguides, fluid transfer channels, and the like.

The type of needle being used may be input into the image guidancesystem, may be a system default, may be detected by a camera or otherdevice, may be received as data from an attached medical device, such assurgical system 349 in FIGS. 3A and 3B, or the information may bereceived in any other appropriate manner. Making the surgical instrumentdisplayed on display 420 resemble the surgical instrument 445 may helpphysicians associate the image guidance data with the real world and mayprovide more familiar guidance information to a physician, therebyfurther aiding the physician in the guidance task. For example, thesurgeon may see the familiar markings on the needle being displayed onthe display 420 and therefore be familiar with the distance and relativeplacement of the displayed needle with respect to other data, such as atumor seen in an ultrasound (not depicted in FIG. 4). This knowledge ofrelative placement of items being displayed may help the surgeon get theneedle into place.

Consider an embodiment in which the image in the display 420 has aneedle depicting the portion of the needle that will perform theablation, for example, the portion that emits the radio or microwaveenergy. If the display 420 also includes ultrasound data, then thedoctor may be able to find the tumor she wishes to ablate by moving theultrasound probe around until she spots the tumor. In variousembodiments, she will be able to see the displayed ultrasound data andits location relative to the displayed needle with the markings. She canthen drive the needle until she sees, on display 420, that theemitter-portion of the needle encompasses the tumor in the ultrasound,also seen on display 420. When she activates the ablation, she can thenbe much more certain that she has ablated the correct portion of thetissue. Various embodiments of this are discussed more below.

As another example, consider the physical markings that may be on theinstruments themselves. These markings can help orient a physicianduring use of the instrument. In some embodiments, the image guidanceunit may represent these markings in the images displayed in thedisplay. For example, certain ultrasound transducers are built with anorientation mark (e.g., a small bump) on one side of the transducingarray. That mark may also be shown in the ultrasound image on thescanner's display, to help the physician understand where the scannedanatomical structures shown on screen are located under the transducer,inside the patient. In some embodiments, the image guidance system maydisplay a symbolic 3D representation of the orientation mark both nextto the motion-tracked ultrasound slice (e.g., moving with the displayedultrasound slice) and next to the 2D ultrasound slice also displayed bythe IVS. An example of this is displayed in FIG. 7, where a smallrectilinear volume corresponding to a feature on an ultrasound probe isshown both in proximity to the ultrasound slice displayed in 3D and theultrasound slice displayed as a 2D image.

Other embodiments will track and display other types of instruments andtheir features. For example, a surgeon may want to track one or more ofa scalpel, a cauterizer (including an electrocauterizer and Bovies),forceps, cutting loops on hysteroscopes, harmonic sheers, lasers(including CO₂ lasers), etc. For example, in various embodiments, thefollowing devices may be tracked and various aspects of their designdisplayed on display 420:

-   -   Olympus™ OES Pro Hystero-Resectoscope, SonoSurg Ultrasonic        Surgical System    -   Olympus™ GF-UC 160 Endoscope    -   Wallus™ Embryo Transfer Catheter    -   AngioDynamics® NanoKnife™, VenaCure™ laser, StarBurst, Uniblade,        Habib®Resector    -   Bovie® Electrodes    -   Covidien Evident™, Cool-tip™ Ablation Antennas, Opti4™        Electrodes    -   Microsulis MEA (microwave endometrial ablation), Acculis    -   Halt™ Medical System    -   Optimed BigLumen Aspiration Catheter    -   Optimed Optipure Stent    -   Central venous catheterization introducer needle (such as those        made by Bard and Arrow)

Once tracked, a physician may be able to see image guidance data ondisplay 420 that will allow her to know the relative pose, location, oremplacement of the tracked instrument(s) with respect to one another orwith respect to imaging data and will be able to see, on display 420,the features of the instrument rendered in the scene.

Depicting Ablation Volume and Other Instrument Information

Various embodiments of the systems herein will depict as part of theimage guidance data information related to the surgical instruments. Forexample, in some embodiments, an image guidance system such as thesystems of FIG. 3A or 3B may illustrate an expected spherical ablationvolume. For example, FIG. 5 shows an ablation needle 545 which has adarkened portion that indicates where the radio frequency or microwaveenergy for ablation will be emitted. In some embodiments, an imageguidance system may display on display 520 the expected ablation volume502. The ablation volume 502 may be shown as a transparent volume, awireframe volume (depicted in FIG. 5), as a point cloud of variousdensities, as an outline, as a volume, or in any other appropriatemanner.

For some ablation needles, the expected volume of ablated tissue isneither spherical nor centered at the tip of the needle. For example: aCovidien surgical microwave needle has an ellipsoidal ablation volume; aCovidien Evident transcutaneous microwave needle has a teardrop-likeablation volume; RFA Medical's bipolar ablation system uses two needlessimultaneously, where each needle has paddles that deploy after theneedle is inserted inside the tissue (which one may equate to a canoe'soar). In some embodiments, the ablation volume for such a needle is, toa first approximation, a volume that lies directly between the paddlesof the two needles.

The position and orientation of the volume may be specified by theplacement of a tracked needle, such as needle 545 in FIG. 5. In someembodiments, with single needle ablation systems, the volume'sapproximate size (e.g., girth and length, if ellipsoidal) may be eitherspecified by the physician, or automatically computed by the guidancesystem. The ablation volume may be based on numerous parameters such asthe needle make and model, power and duration settings of the microwaveor radio frequency generator, measured or estimated temperature andimpedance of the target tissue or other tissue information, a formula, alook-up-table, fixed or default values, or based on any otherappropriate available information.

Other instrument information may also be depicted. For example, if acauterizer is tracked as part of an image guidance system, then thecauterization volume may be determined or estimated and that volume maybe displayed. If a laser is tracked as part of the image guidancesystem, then the projected laser path may be determined or estimated anddisplayed.

Depicting Needle Drive Projection and Other Prediction Information

In certain procedures, there may be prediction information related tothe surgical instruments. In the context of scalpel movement, this maybe the location that the scalpel will hit if a physician continues tomove the scalpel in a particular direction. In the context of ablation,this may be the projected needle placement if it is driven along itscentral axis. FIG. 6 illustrates both an ablation volume for an ablationneedle and a projected needle drive 603. If a physician is driving anablation needle 645 into tissue (not pictured), then she may want toknow where the needle will be driven. In some embodiments, the projecteddrive of a needle 645 may be depicted on the display 620 and may showthe physician the projected path 603 that the needle will take if it isdriven along its central axis.

In some embodiments, in order to aid the physician in placing ororienting a needle 645, an image guidance system, such as that depictedin FIG. 3A or FIG. 3B, may draw a number of rings about the axis of theneedle shaft, extrapolated beyond its tip, as depicted in FIG. 6. Aphysician may view and manipulate the position and orientation of theneedle 645 and its expected drive projection (via its displayedprojected trajectory) before it enters the patient's tissue. In someembodiments, this is accomplished by the doctor positioning the virtualrings in the drive projection such that they are co-incident (or passthrough) the ultrasound representation of a target, such as a tumor thatthe doctor has spotted in the ultrasound. This may allow the physicianto verify that the needle 645 is properly aimed at the target and candrive the needle 645 forward into the tissue such that it reaches itsdesired target or destination. For example, if the doctor spotted atumor in the ultrasound image (not pictured in FIG. 6), she may be ableto align the ablation needle 645 such that the drive projection rings ondisplay 620 intersected or otherwise indicated that the needle, ifdriven straight, will reach the tumor.

The rings may be spaced at regular (e.g., 0.5, 1, or 2 cm) intervals toprovide the physician with visual cues regarding the distance from theneedle tip to the targeted anatomy. In some embodiments, the spacing ofthe rings may indicate other aspects of the data, such as the drivespeed of the needle, the density of the tissue, the distance to alandmark, such as the ultrasound data, or any other appropriate guidancedata or property. In some embodiments, the rings or other trajectoryindicator may extend beyond the needle tip, by a distance equal to thelength of the needle-shaft. This way, the user knows if the needle islong enough to reach the target—even before the tip enters the patient.That is, in some embodiments, if the rings do not reach the target withthe tip still outside the body, then the tip won't reach the target evenwhen the entire length shaft is inserted into the body.

Other display markers may be used to show trajectory, such as a dashed,dotted, or solid line, transparent needle shaft, point cloud, wireframe, etc. In some embodiments, three-dimensional rings may be used andprovide depth cues and obscure little of the ultrasound image. Virtualrings or other virtual markers may be displayed semi-transparently, sothat they obscure less of the ultrasound image than an opaque markerwould.

Other prediction information may also be displayed. For example, if ascalpel is being tracked by the image guidance system, then a cuttingplane corresponding to the scalpel may be displayed (not pictured). Sucha cutting plan may be coplanar with the blade of the scalpel and mayproject from the blade of the scalpel. For example, the projectedcutting plane may show where the scalpel would cut if the doctor were toadvance the scalpel. Similar prediction information may be estimable ordeterminable for cauterizers, lasers, and numerous other surgicalinstruments.

Depicting Combinations of Graphics

As discussed herein, when there are multiple instruments or devicesbeing used in a procedure, images, graphics, and data associated withthe multiple instruments may be displayed to the physician. In someembodiments, as depicted in FIG. 7, when there are two devices 745 and755 being used and tracked in a procedure, data, images, and graphicsassociated with those two images may be combinable and may be displayedon the same display. FIG. 7 depicts an ablation needle 745 and anultrasound probe 755 being used during a procedure. Data associated witheach of the devices 745 and 755 are displayed on the display 720.

The data from two or more devices may be combined and displayed based ontheir relative emplacements or poses. For example, an ultrasound image704 may be displayed with respect to an ablation needle on a display 720in a manner that estimates the relative emplacements or poses of anultrasound probe 755 and ablation needle 745. This is depicted in FIG.7. In FIG. 7, the graphics associated with the ablation needle 745,including the ablation volume and projected drive location are shownspatially located with the oriented planar ultrasound image on display720. In this image 704, a tumor appears in the ultrasound image and theablation needle is shown driven through the tumor. The ablation volumeestimates where ablation would occur if the tissue were ablated at thattime. The physician can see that the ablation volume appears to coverthe tumor displayed in the ultrasound image.

Various embodiments include other combinations of graphics. For example,in some embodiments, data related to a single surgical instrument (suchas an ablation needle, ultrasound probe, etc.) may be presented in morethan one manner on a single display. Consider an embodiment in whichdevice 745 is an ablation needle and device 755 is an ultrasoundtransducer. If a physician orients ultrasound transducer 755 such thatit is perpendicular to the monitor, the 3D view of the ultrasound imagewould show only the edge and the ultrasound image would not be visible.In some embodiments, the image guidance system could track thephysician's head using a position sensor, such as first and/or secondposition sensing units 310 and/or 340 of FIG. 3A or FIG. 3B. Thephysician then may be able to move her head to the side, so that shesees the ultrasound image from a different perspective.

In some embodiments, the image guidance system can constantly display anadditional 2D view of the ultrasound image 705 (in screen space),simultaneous to the 3D depiction of the procedure, so that theultrasound image is always visible, regardless of the orientation inwhich the physician holds the transducer. This is illustrated in FIG. 7.This display of the ultrasound data may be similar to what a physicianis accustomed to seeing with traditional ultrasound displays. This maybe useful to provide the physician with imaging to which she isaccustomed and allows a physician to see the ultrasound data regardlessof the then-current orientation of the ultrasound probe with respect tothe user.

In some embodiments, the 2D view 705 of an ultrasound image is depictedin the upper right corner of the monitor (though it can be placed in anycorner). The guidance system can automatically (and continually) choosea corner in which to render the 2D view of the ultrasound image, basedon the 3D position of the surgical instruments in the rendered scene.For example, in FIG. 7, ablation needle 745 may be held in thephysician's left hand and the needle shaft is to the left of the 3Dultrasound image slice, so that the 2D ultrasound image 705 in the upperright corner of display 720 does not cover any of the 3D features of theneedle (or vice-versa). If the needle were held in the physician's righthand, the virtual needle shaft would appear on the right side. Toprevent the 2D ultrasound image in the corner of display 720 fromcovering the needle shaft, the system can automatically move it to acorner that would not otherwise be occupied by graphics or data.

In some embodiments, the system attempts to avoid having the 2Dultrasound image quickly moving among corners of the display in order toavoid overlapping with graphics and data in the display. For example, afunction ƒ may be used to determine which corner is most suitable forthe 2D ultrasound image to be drawn in. The inputs to ƒ may include thelocations, in the screen coordinate system, of the displayed needle tip,the corners of the 3D ultrasound image, etc. In some embodiments, ƒ'soutput for any given point in time is independent of ƒ's output in theprevious frames, which may cause the ultrasound image to move amongcorners of the display rapidly. In some embodiments, the image guidancesystem will filter ƒ's output over time. For example, the output of afilter g, for any given frame, could be the corner which has been outputby ƒ the most number of times over the last n frames, possibly weightingthe most recent values for ƒ most heavily. The output of the filter gmay be used to determine in which corner of display 720 to display the2D ultrasound image and the temporal filtering provided by g may allowthe 2D ultrasound image display to move more smoothly among the cornersof the display 720.

In some embodiments, other appropriate virtual information can beoverlaid on the 2D ultrasound image as well. Examples include: anindication of the distance between the needle's tip and the point in theplane of the ultrasound image that is closest to the needle tip; thecross section or outline of the ablation volume that intersects with theultrasound slice; and/or the intersection point, box, outline, etc.between the needle's axis and the ultrasound image plane.

Representing Spatial Relationships

At times, when three dimensional relationships are depicted in 2D, oreven in 3D, it may be difficult to gauge the relative positions,orientations, and distances among various objects. Consider FIG. 7, inwhich an ablation needle is shown intersecting an ultrasound image.Depending on the embodiment, it may be difficult to determine therelative angle of the ablation needle and the ultrasound image as wellas the distances of various portions of the image to the ablationneedle.

In some embodiments, the image guidance system may indicate spatialrelationships with graphical indicators. For example, in FIGS. 8 and 9,graphical indicators help indicate the spatial relationship between aneedle and an ultrasound image. These also provide an indication of therelative angle of the needle and the image.

In some unpictured embodiments, the image guidance system may draw“guidance graphics” in the form of projective lines between the needleand the ultrasound slice. These lines may be perpendicular to the planeof the slice and serve to indicate the most likely location in the slicewhere the needle will become visible if it is moved to become coplanarwith the slice. Together with stereoscopic head-tracked visualization,the projective lines help a physician determine a more accurateassessment of the location of the needle with respect to the ultrasoundslice.

Returning to FIGS. 8 and 9, in some embodiments, uniform-thickness linesbetween virtual needle and slice plane may be displayed on display 820and 920. The lines may represent the spatial relationship withthree-dimensional rectangular (or any shape) needle projection bars. Invarious embodiments, the projection bars may be drawn perpendicular tothe image, and in such a way that their small edges are aligned with (orparallel to) either the vertical (FIG. 8) or the horizontal (FIG. 9)margins of the ultrasound slice. In some embodiments, the screen-spacesize of the projection bars may be variable (e.g., distance-dependent)due to perspective. Thus they may provide depth cues for the physician.Further, the staircase appearance of the bars' end edges at the plane ofthe slice may be a further visual cue for the orientation of the needlewith respect to the slice.

In some embodiments, when the needle is nearly perpendicular to theultrasound image, the projection bars may appear similar to the needleitself. Therefore, in some embodiments, the rectangular projection barsmay not be displayed when the needle is nearly perpendicular to theultrasound image plane. Instead no projection information may bedisplayed or project lines may be displayed as dotted or dashed lines.The display of projection lines is illustrated in FIG. 10. In someembodiments, as depicted in FIG. 10, a line may also be drawn that is aprojection of the needle onto the image plane of the ultrasound image.This may provide relative orientation information to the user orphysician.

Reducing Stereo Display Artifacts with Object Choice

Stereoscopic displays separate the imagery shown to the user's eyes invarious ways. Cathode Ray Tube (CRT) based devices, may use temporalsequencing, showing imagery for the left and right eye in temporalsequential alternation. This method may also be used by newer,projection-based devices, as well as by 120-Hz-switchable liquid crystaldisplay (LCD) devices. Another type of stereoscopic display uses spatialseparation such as alternating rows (AR) or alternating columns (AC).Example AR displays include the Miracube G240S, as well as Zalman TrimonMonitors. AC displays include devices manufactured by Sharp, as well asmany “auto-stereoscopic” displays (e.g., Philips).

Both AR and AC monitors have reduced (often by at least 50%) resolutionin one dimension: vertical for AR and horizontal for AC. As a result,some elements—most of all thin lines—when displayed as nearly horizontalAR units and nearly vertical on AC units often feature noticeableartifacts such as aliasing and discontinuities (e.g., a continuousnear-horizontal line may appear dashed on an AR display). Theseartifacts may have a negative impact on stereoscopic fusion (e.g., thehuman brain's ability to merge the separate left and right eye imagesinto a single 3D mental representation).

Stereoscopic fusion may be useful for improved perception and needleguidance by a physician. In some embodiments, an image guidance system,such as system 300 in FIG. 3A or FIG. 3B, may use thicker lines,particularly in the horizontal, and use fewer near-horizontal lines,borders, and structures when using AR displays. In some embodiments,when using an AC display, the image guidance system may use thickerlines, particularly in the vertical, and fewer near-vertical lines andstructures.

In some embodiments, the projection markings such as rectangular barsshown in FIGS. 8 and 9 are chosen in part based on the type of displaybeing used. Representing the projections in thicker form, such asrectangular bars may help overcome the limitations of AR and ACdisplays. Further, in some embodiments, the projection bars can beoriented along their long axes (e.g., axes that are perpendicular to theslice) such that in the case of an AR display their short end edges arealigned with the vertical edge of the ultrasound slice and thus willappear mostly vertical in the stereoscopic image. This is illustrated inFIG. 8. For AC displays, in some embodiments, the bars are oriented suchthat their ends are parallel to the top and bottom of the ultrasoundslice and thus are more likely to appear near-horizontal in thestereoscopic image. This is depicted in FIG. 9. The projective bars wereused here as an example, but this technique can be applied to anydisplay element in order to accommodate AR or AC displays. For example,if needle drive projections are displayed in an embodiment, such as thatdepicted in FIG. 6, then the shape chosen to indicate the projectedneedle drive may be adapted (by showing spheres instead of circles,e.g., or by expanding the width of the drive indicators) to take intoaccount that an AR or AC display is being used.

Reducing Stereo Display Ghosting Effects

In some embodiments, stereoscopic displays may suffer a “ghosting”phenomenon, which may be crosstalk between the left and right eyeimages. Ghosting can affect frame-sequential, AR, or AC displays.Ghosting may be exacerbated when there is a great difference between thecolors or brightnesses between the left and right eye imagery shown in aregion of the displayed image. Due to the nature of stereoscopic imagedisplay, these differences may occur where the (virtual) stereoscopicdepth of the 3D scene to be rendered varies from the plane of thedisplay surface.

In some embodiments, the image guidance system modifies the color andbrightness of display elements that are in front of or behind the planeof the display (e.g., where the needle and ultrasound image intersect orthe plane of the monitor). The image guidance system may shift therendered color towards a uniform or background color with increasingdistance from the display plane. In some embodiments, this may beaccomplished by means of the OpenGL “fog” feature, which can “bathe” alldisplayed geometry in a color whose opacity increases with distance fromthe display plane. This may vary on a per-pixel basis. The farther theobject is behind the display plane, the more it may be blended with thebackground color. This may also be applied to objects in front of thedisplay plane by reversing the depth or Z coordinates. In someembodiments, ghosting reduction may also be implemented as a fragmentshader or other routine or program, running on programmable graphicshardware or a CPU, etc. The input to a fragment program may be the colorof the pixel, the color of surrounding pixels and the depth (e.g., Zdepth, or the absolute distance to the plane of the monitor). Theprogram may use the first two inputs to compute the contrast in theregion local to the current pixel. The program may then reduce thecontrast for those high-contrast regions, based on how far they are fromthe monitor's display plane. This program may also be implemented as theconverse or opposite of an edge enhancement filter while also takinginto account the screen depth of the edges.

Representing Non-Intersecting Objects or Images

When data related to two devices or surgical instruments are displayedwith relative emplacement, it can be difficult to orient their relativelocations if they do not intersect. In some embodiments, an imageguidance system will render relative location information. The relativelocation information may be shown with color (e.g., objects may berendered in brighter colors if they are closer), with renderingtechniques (e.g., objects may be rendered with transparency so that oneobject behind another may be visible, but visually appear behind thecloser object), with geometry (e.g., a geometric connector may be shownthat will allow the viewer to discern the relative relationships), orwith any other appropriate technique. FIGS. 11 and 12 illustrate examplegeometry and transparency being used to show relative locations of twoobjects.

For example, in some embodiments, if the intersection point of anablation needle is outside of the area of the ultrasound slice, theimage guidance system can draw geometry, such as a line (or rectangle)in the plane of the slice to indicate the needle's and ultrasoundimage's relative positions. This is depicted in FIG. 11. In someembodiments, the relative locations could also be represented usingvertical and horizontal elements coplanar with the ultrasound or otherimage slice, as depicted in FIG. 12. In some embodiments, using geometrythat is coplanar with the ultrasound image slice may provide anintuitive understanding of the relative locations of an image slice andan ablation needle.

Rendering Techniques for 3D Fusion

In some embodiments, various data displayed by the image guidance unitmay be displayed as lines, frames, or 2D objects. For example, theablation volume of FIG. 5 may be rendered as a wire-frame using 2Dlines. Similarly, the projection lines of FIG. 8 may be rendered as 2Dlines.

In some embodiments, some or all of the displayed data may berepresented in 3D space and rendered as such. For example, the ablationvolume of FIG. 5 may be represented using beam-shaped, pipe-like, sweptrectangular, 3D polygonal primitives, etc. in order to “build” thewireframe representation. Similarly, the rectangular projection lines ofFIG. 8 may optionally be built with beam-shaped, pipe-like, sweptrectangular, polygonal solids, etc. For example, the projection lines ofFIG. 8 may be flat rectangular prisms, and the ablation volume of FIG. 5may be represented as thin, curved tubes with a colored stripe pattern.In some embodiments, using 3D objects to represent various data mayprovide improved 3D fusion for viewers.

Additionally, in some embodiments, a “surface detail” texture may beadded to various objects. Adding surface detail may aid with stereofusion because of the addition of surface texture that may providestereoscopically “fusible” details (e.g., anchor points) on an object. Asimple line or uncolored 2D object may not provide as many anchorpoints. Examples of possible textures include the use of color stripesor mosaics, metallic textures, and random noise textures. In someembodiments, textures may be selected so that the spatial pattern andfrequency does not cause aliasing in the stereoscopic display'salternating scanlines or columns, nor with the checkerboard-interleavedpixels which are used by certain projection-based stereoscopic displays.

In some embodiments, surface shading from one or more light sources isused. Examples of surface shading that may be used includes surfaceshading from one or more light sources, which may be supported ingraphics processor hardware, as well as other enhancements like castshadows, and cues such as global illumination. Surface shading maycontribute to increased depth perception in the guidance image.

Marking Points of Interest

In certain procedures, physicians need to keep track of multiple spotswithin the volume of the patient or keep track of a single point orfeature while looking at other parts of the volume. For example, when aphysician is going to perform an ablation, before inserting any needles,the physician will often scan the tissues at the procedures site to findall targets (e.g., tumors) and note other features of the tissues. Then,later in the procedure, the physician may return to the previouslyidentified points-of-interest. For example, a physician might first scanthe liver and find seven lesions that she will attempt to ablate. Afterablating the first lesion, she might be required to find the secondlesion again, and so forth. Before finishing the procedure, she might berequired to verify that she has ablated all seven of the lesions thatshe identified at the beginning of the procedure. This constant scanningand rescanning can be time consuming and error prone. Further, in aprocedure where the surgeon is attempting to locate, for example,fluid-filled cysts, once a needle pierces the cyst, the fluid may drainout, making the target difficult or impossible to locate again withultrasound.

In some embodiments, the image guidance system may allow the physicianto mark or keep track of points or features of interest. In variousembodiments, the physician can mark the points or features of interestin various ways. For example, consider a procedure where the doctor isusing the image guidance system with an ablation needle and anultrasound probe. The doctor may be able to mark the point by pressing abutton on a keyboard or medical device, by gesturing or issuing a verbalcommand, or with any other appropriate method. The point of interest maybe marked at the point where the needle intersects with the ultrasoundimage plane, where the needle's projection intersects with theultrasound image plane, or any other appropriate relationship (such asat the location of the tip of the needle). For example, when thephysician identifies a point-of-interest 1301 within the ultrasoundimage, she can point to it using the needle even if the needle isoutside the body of the patient. This is depicted in FIG. 13. Thephysician (or assistant) may then press, for example, a button or footpedal, which informs the image guidance system to store the 3D positionof this point-of-interest 1301. FIG. 14 illustrates an X being displayedwhere a point of interest 1401 has been marked. In some embodiments, thesystem may then display the position of this point-of-interest 1401relative to the ultrasound plane and the needle. For example, anX-shaped marker 1502 may be displayed on display 1520 to show therelative position of the marked position and the surgical instruments,as depicted in FIG. 15. In some embodiments, the system may also displaya bar that connects the X marker 1502 of the point-of-interest to thenearest point (or the point to which a normal vector of the image planewould reach the X), as depicted in FIG. 15. This visually indicates, tothe physician, the distance between the ultrasound image and thispoint-of-interest. Should the physician want to see the point ofinterest again in the live ultrasound image, the graphics indicate towhere she should move the ultrasound transducer to view that point inthe ultrasound image. In some embodiments, the image guidance system mayalso display the numerical distance (e.g., in mm) between the ultrasoundimage and the point-of-interest (not shown).

Physicians, during some liver ablation procedures, may manage fifteenpoints-of-interest, or even more. As depicted in FIG. 15, in someembodiments, there may also be multiple markers 1502 of point ofinterest simultaneously displayed. The image guidance system may be ableto store and display any number of points of interest simultaneously. Ifthere is more than one point-of-interest in view, the image guidancesystem may display a number next to each one (not pictured). In someembodiments, in order to reduce visual clutter if there are many pointsof interest, those points which are closer to the ultrasound image planeare drawn more saliently or vividly (with more bold color and thickerlines) while the points that are far away are drawn less saliently (moretransparent, blurred, muted colors, etc.). Additionally, in variousembodiments, other representations other than an X (such as a point,point cloud, sphere, box, etc.) may be used and multiple markers orlocations may be represented with different markings.

In some embodiments, the image guidance system stores thepoints-of-interests' positions in the position sensing system'scoordinate system. If the position sensing system is fixed to the imageguidance system, then, if the patient or image guidance system aremoved, stored points-of-interest may become incorrectly located. In someembodiments, this can be remedied via a fiducial or other detectablefeature or item, the pose of which relative to the tracking system maybe continually, continuously, periodically, or occasionally measured.The fiducial may be attached to the operating table, the patient's skin,or even embedded into the tissue itself (e.g., as a magnetic trackingcoil), and the points-of-interest’ positions, relative to it, can bestored and displayed. For example, in a system where magnetic trackingis used, a magnetic tracking coil may be affixed to the operating tableor patient.

Data Visualization Processes and Data

Displaying Volumetric Data

As discussed elsewhere herein, there are numerous types of volumetric or3D data that various embodiments of the image guidance system herein maydisplay. Such data may include CT scans, MRI, PET, 3D ultrasound, andany of numerous other types of 3D data. In some embodiments, in order todisplay 3D data on a 2D interface, such as a computer screen, or even a3D interface, such as a head-mounted display or other 3D display, asubset of the data is chosen to display. This subset of data can includeaxis-aligned slices, the entire volume, or a sub-volume of the data. Aninherent difficulty with image guidance is the display of threedimensions of data on a two dimensional screen or “dual eye” threedimensional display. When displaying 3D data, such as CT scans, a systemmight only display a single plane, or show three orthogonal planesseparately on the screen, as shown in FIG. 1. The data may also be shownas a volumetric ‘block’ of data, as shown in FIG. 2.

Therefore, as depicted in FIG. 2, if a sub-volume is displayed, theouter surface of the volume may be displayed as a box on the screen andthe data interior to the rendered box may not be visible to the doctor.The 3D data may be displayed in correspondence to the surgicalinstrument that the doctor is holding (e.g., note the needle and 3D datainteracting in FIG. 2). The 3D data may also be displayed asaxis-aligned slices, as shown in FIG. 1. In previous systems, when thedoctor wanted a different view of the 3D data (either a differentvolume, as in FIG. 2, or different set of images, as in FIG. 1), shewould have to alter the location of the displayed planes, using, forexample, a cursor (controlled by a mouse or trackball).

In some embodiments, the image guidance system controls the display ofthe 3D data with a medical device. For example, as depicted in FIG. 16,one of the planes 1603 of the 3D data displayed on display 1620 may bealigned with a tracked surgical needle, such as surgical needle 1645.Additionally, in some embodiments, separate data related to anothermedical device, such as ultrasound data, may be displayed on display1620 (not shown). This data may be obtained, and its pose or emplacementknown, based on a tracked ultrasound probe, such as tracked ultrasoundprobe 1655 of FIG. 16.

In some embodiments, the displayed plane 1603 of the 3D data may beaxis-aligned in one direction and controlled by the surgical instrumentin the other two directions. This may result in an image such as thatdisplayed in FIG. 16, where the image is aligned with the X, or Y axis,and the location in the X and Y axes is controlled by the needleplacement. The plane of orientation of the surface may also be chosenbased on alignment with the plane of the display (choose the closestplane to the screen's plane); closest to one of the three traditionalorthogonal planes (sagittal, transverse, coronal); or one specified bythe physician (or an assistant who is not in the sterile field), using acontrol knob, lever, or foot pedal. If the needle is curved (or bentinto a curve from the pressures acting on it by the tissues and thephysician's hand), the system may choose the plane that most clearlydisplays the curvature of the needle.

In some embodiments, the display plane may also be controlled by therotation of the medical device. This is depicted in FIG. 17, whichillustrates that, once the needle 1745 is rotated, the display plane forthe volumetric data is rotated and shown in display 1720. In someembodiments, the image guidance system may choose a plane thatintersects some feature or landmark on the needle's handle (such as theport where the wire exits, or a molded ridge) or the needle's shaft(such as the openings in the shaft from which the paddles deploy on theneedles made by RFA Medical™). Then, when the physician physicallytwists the needle, it has the effect of rotating the cross-sectionalplane (FIG. 4). This may allow a physician to intuitively specify itsorientation by using the needle. The rotational arrows in FIG. 17, invarious embodiments, may or may not be shown on display 1720. Additionalembodiments are given in FIGS. 23 to 26 and accompanying text.

The system can work with static or real-time (or near real-time)volumetric data. In some embodiments, the system may display a crosssection of the volumetric image along a plane that intersects the axisof a surgical needle or other surgical or medical device. The system maycontinually update the position and orientation of the cross-sectionalplane as the physician moves the needle. If the volumetric or 3D imagedata is from a real-time imager, such as a 3D ultrasound transducer,then instead of having to continually manipulate both the needle andtransducer, the physician may place the transducer such that theultrasound volume includes the target, and then to leave the transducerstationary. She then can manipulate the needle's position, before itpierces the tissues, until the target tissue appears visible in theslice controlled by the needle. (See, e.g., FIGS. 32-34 and accompanyingtext).

Physicians often attempt to maintain 2D ultrasound planes in order tokeep the shaft of the needle within the ultrasound image. Doing so mayallow then to watch as the needle advances through the tissues. In someembodiments herein, the image guidance system may maintain the displayedultrasound image within the needle's path automatically. If the doctorcan see what is in the needle's path (as shown in the displayed plane ofthe 3D volumetric data) she may be able to see what will be in theneedle's path when she drives the needle. By being able to see this, shemay be able to avoid piercing any tissue that should not be pierced bythe needle's path. In some embodiments, as the physician advances theneedle towards a target, the cross-sectional plane may be chosenautomatically by the image guidance system such that it shows theneedle, the tissue surrounding the needle, and any structures that areabout to be pierced by the needle. (See, e.g., FIGS. 23 to 36D andaccompanying text)

In some embodiments, similar techniques can be used to control theimages that are displayed in 2D, as in FIG. 1. That is, the manipulationof a medical device, such as a needle, scalpel, ultrasound probe, etc,can control what planes are shown in the display. For example, if asurgeon moved a needle toward and away from herself, she might sweepthrough the 3D volume and the corresponding slices of the 3D volume maybe shown on the display. (See, e.g., FIGS. 23 to 36D and accompanyingtext)

As noted above, various embodiments use live or real-time volumetricimages (e.g., intraoperative 3D ultrasound), static volumetric images(e.g., pre-operative CT or MRI) or hybrid volumetric images (e.g.,pre-operative CT images that are continuously warped to be inregistration with live 2D ultrasound or fluoroscopic images, orlaser-scanned 3D point sets representing the surface of tissue).

Visualizing Portions of Volumetric Data

When displaying 3D volumetric data, voxels in front (closer to thevirtual camera) typically obscure the voxels behind them. This hidesinformation that may be important from preoperative 3D data andreal-time or live 3D data because the surgeon can only clearly view theclosest voxels. As noted above, one way to deal with displayingvolumetric data is to allow the doctor to view the data as 2D slices, incross section, etc. In some instances, however, there are determinabledifferences among the voxels in the 3D data. Therefore, in someembodiments, the image guidance system can display only those voxelsthat meet certain criteria. For example, in the case of a preoperativeCT scan, the voxels containing bone matter should be determinable basedon tissue density. Therefore, the image guidance system may display onlythose voxels within a certain range of tissue densities. As anotherexample, when the volumetric image of a fetus in the womb is visualizedfrom 3D ultrasound data, embodiments of the image guidance system maymake all voxels that represent fluid surrounding the fetus transparentor invisible, thus allowing the surface of the fetus to be visible.

Some types of 3D imaging data can provide flow information. In someembodiments, the image guidance system can be set to only display onlythose voxels that contain flow information. For example, some ultrasoundscanners, including 3D ultrasound scanners, can measure motion and flowwithin the imaged area using Doppler techniques. The portions of theimage that have flow above some threshold velocity may be displayedusing a particular color or a gradient of colors determined based on theflow information. The remainder, non-flowing part of the ultrasoundimage may be drawn as traditional grayscale, or may be made invisible.

For example, in some embodiments, the image guidance system may have aDoppler 3D mode, in which volumetric images (such as a 3D ultrasound)are sampled, and then those volumetric images are displayed such thatonly those voxels which contain flow (above some threshold velocity) areopaque, while all other voxels (without sufficient flow) are madetransparent. By displaying only the portions of the image that haveDoppler-detected motion, the image guidance system may provide aneasy-to-decipher 3D image of the progress of the ablation. For example,FIG. 18 illustrates a needle 1845 ablating tissue while a 3D ultrasoundprobe 1855 is collecting ultrasound data. The Doppler data is collectedfrom 3D ultrasound probe 1855 and only the progress of the ablation 1804is shown on display 1820. One or more slices of the collected ultrasounddata may also be shown on display 1820 (not pictured).

In some embodiments, Doppler information can be collected over time asthe doctor sweeps a 2D ultrasound probe over a volume. Since theultrasound probe is tracked, the image guidance system can determine therelative locations of the collected ultrasound slices and locate them in3D space. From this data, in some embodiments, the image guidance datacan approximate 3D flow information in various ways. For example, insome embodiments, in order to observe the progression of the ablationwith a 2D transducer, the physician may continually sweep the ultrasoundtransducer back and forth over the general area of tissue that containsthe lesion and ablation needle. Some of the tissue being ablatedcontains may expand into micro bubbles that can be detected inultrasound. The image guidance system may extract those pixels andrepresent the area of Doppler flow (e.g., “a Doppler slice”), relativeto the latest 2D ultrasound image (“the ultrasound slice”). For example,as depicted in FIG. 19, as 2D ultrasound probe 1955 is swept across thevolume of interest, Doppler slices 1921 may be collected and displayedon display 1920. Additionally, the ultrasound slice 1922 may also bedisplayed. Optionally, ultrasound needle 1945 may also be displayed ondisplay 1920.

In some embodiments, older Doppler slices may be drawn moretransparently, with more blur, in order to reflect that, the older aslice is, the more out-of-date its image contents have become.Eventually, every sampled slice may become completely invisible, nolonger being presented to the user. This prevents out-of-date imagesfrom obscuring the view of more recent images.

In some embodiments, the ultrasound slices are rendered using varioustechniques. For example, they might be rendered using a technique fromGarrett et al., Real-Time Incremental Visualization of DynamicUltrasound Volumes Using Parallel BSP Trees. Proc. IEEE Visualization'96 (San Francisco, Calif., Oct. 27-Nov. 1, 1996), pp. 235-240, 490,which is hereby incorporated by reference for all purposes. For example,each rendering frame, a binary spatial partition (BSP) tree datastructure may be used to compute a back-to-front ordering of each slice.Then the slices may be rendered in back-to-front order, such that thetransparency and occlusion are handled correctly. Reconstruct, in 3D,only the portions of the image that have Doppler-detected motion maymake the 3D images easier for the physician to decipher, therebyimproving her understanding of the progression of the ablation.

In some embodiments, a rendering technique is used to sort Dopplerslices using the depth of the center-point of each Doppler slice. Thistechnique may result in an approximate back-to-front ordering as theslices may intersect. In some embodiments, a BSP tree algorithm maysplit one slice into two in the case where they intersect each other.Since sorting by the slices' center point depths results in anapproximate ordering, the resulting rendering may have some visualartifacts (e.g., pieces of one slice that should appear to be behindanother slice, but instead appear in front of it). To minimize thepresence of these artifacts, in some embodiments, the image guidancesystem may render the Doppler slices in two separate passes. In thefirst pass, a clipping plane, co-incident with the most recentultrasound slice, is employed to discard the portions of any Dopplerslices that are in front of the ultrasound slice. Then the ultrasoundslice is drawn, followed by the second pass of the Doppler slices. Thissecond time, the clipping plane may discard portions of any Dopplerslices that lie behind the ultrasound slice.

Thin Visualization of 3D Data

As noted above, in traditional rendering of 3D volumetric data, data orvoxels in the front of the rendered image may occlude data or voxelstowards the back of the volume. This may be a problem when the data thatis further back from the surface is the information that a physicianneeds to see. Various techniques for overcoming this are given above.(See, e.g., FIGS. 23 to 36D and accompanying text). More techniques aregiven in this section.

In some embodiments, the 3D volume data herein may be rendered with a“thin” field of view or depth of focus. For example, in someembodiments, the image guidance system renders a singleplane-of-interest in sharp focus, while rendering the rest of the volumedataset, in perspective projection, as transparent and blurry, withstereo cues and/or motion parallax, and spatially registered to theplane-of-interest. This provides the user some context andrepresentation for features located outside of the plane-of-interest,while minimizing their visual interference with image features in theplane-of-interest. In some embodiments, a thin volume of interest (asopposed to a plane of interest) may also be rendered. The volume ofinterest may include a small and/or user-controllable slice of data thatis rendered in sharp focus with, as above, the rest of the volumetricdata (in front of and behind) the thin volume of interest rendered in ablurry, transparent, or other technique.

This thin depth-of-field volume visualization may have several medicalapplications. It may be useful to help the physician/user guide a needletowards a target located in the plane-of-interest, while simultaneouslyavoiding features in front of or behind the plane-of-interest. It mayalso be used to identify and mark features (e.g., points, organboundaries, tumors) in the volumetric images (described above). Thesetasks can be performed with real-time volumetric images (e.g.,intraoperative 3D ultrasound), static volumetric images (pre-operativeCT, MRI, etc.) or hybrid volumetric images (e.g., pre-operative CTimages that are continuously warped to be in registration with live 2Dultrasound or fluoroscopic images, or laser-scanned 3D point setsrepresenting the surface of tissue).

Further, this technique can be combined with other techniques herein.For example, in order to control the plane or volume of interest(location, orientation), a surgeon may manipulate the needle asdescribed above. In some embodiments, the plane or volume of interestmay be parallel and coincident with the screen-plane, or it may havesome other spatial relationship to the surface of the display screen.For example, the plane or volume of interest may contain the needle thatthe physician is placing into the tissue. In some embodiments, thedoctor or other user may interactively manipulate the spatialrelationship of the plane or volume of interest relative to the volumedataset, using the needle, or by controlling a knob, mouse, joystick,etc.

In some embodiments, the thin depth-of-field volume can be displayedsuch that it is superimposed and spatially registered with organ/bloodvessel/tumor surfaces or contours, radiation dose iso-contours, needleguidance information, or any other combination of relevant knownpolygonal or volumetric 3D data.

In some embodiments, when used with stereoscopic monitors, thindepth-of-field rendering can also be used to reduce ghosting (anundesired cross-talk between the two separate images for each eye. Forexample, a high-contrast line in the left-eye image may be slightlyvisible in the right-eye image). When used to reduce ghosting, thevolume or plane of interest may be co-incident or nearly co-incidentwith the screen plane in 3D space (e.g., the surface of the displaymonitor).

In some embodiments, the volumetric data is sliced (e.g., resampled)into a set of image planes or image volumes that are parallel to theplane of interest. (See, e.g., FIGS. 36A-36D and accompanying text) Thedistance between the image planes or volumes may be dependent on theresolution of the volume dataset, the display monitor's resolution, andthe computational resources available (e.g., larger spacing may resultin faster rendering and a higher frame rate, but a lower fidelity imagein each frame). For example, an image plane (or volume) may be createdfor each depth resolution in the volumetric data, or a predefined ordetermined number of slices may be used. Each image plane or volume isthen blurred; the “radius” of the blur may increase with the distancefrom the image slice to the plane or volume of interest (e.g., imagesslices further from the plane of interest may be made blurrier thanimage slices close to the plane-of-interest). The image plane coincidentwith the plane-of-interest itself may have no blur (e.g., it may berendered using the standard reconstruction for the display monitor'sresolution). In some embodiments, the image planes' brightnesses,contrasts and/or transparencies may then be modulated by their distancefrom the plane or region of interest. In some embodiments, the planesmay then be rendered in back-to-front order. Various embodiments mayalso be implemented directly on programmable graphics hardware,dedicated hardware, etc. to reduce processing time and or memory usage.

In some embodiments, in order to reduce computational and memorydemands, the image planes may be spaced such that the further they arefrom the plane-of-interest, the larger the distance between them. Thoseportions of the volume dataset that are farther away from theplane-of-interest, and thus displayed as blurrier and more transparent,will have a lower density of image-planes that sample them.

As another example, consider an embodiment in which the volume isdivided into parallel slices. These slices are parallel to the displayscreen, or parallel to the ultrasound transducer. The slices are drawnfrom back to front. The transparency of the slice is determined by somefunction whose input is its distance from the region of interest (suchas the ultrasound transducer). Similarly the blurriness may bedetermined by a similar function. The transparency may be passed as thealpha parameter to the 3D rendering library (OpenGL, DirectX, etc.).Blur may be passed to the 3D rendering library as a maximum MIPMAPlevel.

The annotations may be drawn in a similar way. Each point of linesegment may be made transparent and blurry based on its distance to theregion of interest, and it's rendering controlled by the alpha and maxMipMap level that is passed to the 3d rendering library.

Some embodiments may involve using pixel shaders (e.g. GLSL, Cg, etc.),that operate over the entire volume image, rather than dividing it intoparallel slices. For each screen pixel, a ray may be cast through thevolume. The color of the screen pixel is determined by accumulating thecolor at each sampled distance from the plane-of-interest. The color foreach sampled distance from the plane-of-interest is the color of thevoxels in the mipmapped 3D texture (which is chosen based on the blurlevel for that distance from the plane-of-interest), and thetransparency for that distance from the plane-of-interest.

Regardless of the rendering implementation, in various embodiments, thevolume can be displayed from several different perspectives:

-   -   From that of the physician, using a position sensor on the        ultrasound transducer and optionally on the physician as well;    -   From that of the camera, x-ray radiation emitter, or imager;    -   From that of the ultrasound transducer;    -   From that of the needle or ablation device.

Tracking and Calibration

Image guidance systems provide real-time guidance to a medicalpractitioner during medical procedures. Numerous examples andembodiments of this are given herein. Image guidance systems requiretracking. In order to track, there is typically a tracking “source” anda tracking “receiver,” although there are many other arrangements knownto those skilled in the art and discussed herein. Examples of trackingare discussed throughout herein and with respect to instruments 345 and355 and tracking systems 310 and 340 in FIGS. 3A and 3B.

In some embodiments, in order to track a device, some portion of thetracking system may be attached to the device. In some instances thismay actually be a source, receiver, fiducial, etc. In optical tracking,for example, a tracking device is employed by the system to continuallyreport the position and/or orientation of tracking fiducials that areattached to the devices to be tracked. In some embodiments, thesefiducials are rigidly affixed to the needle or to its handle. Withknowledge of the geometry of the needle, relative to the fiducials, animage guidance system can compute the position of the needle and itstip.

As noted above, each time a medical practitioner uses a new needle witha guidance system, she might be required to rigidly affix the trackingfiducials to the needle and measure the position of the tip of theneedle, relative to the fiducials. This is an extremely time consumingprocess. She might be required to first tighten screws, or to thread theneedle through a hole or tube. Then she might be required to manuallymeasure the needle length with a ruler (because needle lengths may varyeven for standard needles), and then enter this information into aworkstation. She may also be able to use a dedicated calibration rig,and perform a lengthy, often minutes-long calibration process. The sameprocess occurs for other types of tracking systems as well.

Simplifying Calibration

In order to simplify the calibration process, in some embodiments, theimage guidance systems can utilize something that will indicate theneedle's tip relative to a known or determinable location. If the needleis being tracked (even if not calibrated) when this is done, then theneedle's tip relative to its own tracking fiducials can be calculated.For example, consider a needle's fiducial mount 2000 comprising aspring-loaded plastic clip 2010, with a groove embedded in the innersurfaces of both of the inner sides of the clip. In some embodiments,the tracking fiducials 2040 may be attached to a fixed piece of theclamp. The user may attach the needle by first pressing onto the sideopposite the fulcrum to open the “jaws” (as depicted in FIG. 20). Shemay then insert the needle 2020 into the groove on the fixed part of theclamp while maintaining pressure upon the upper sides, above the pivotpoint, of the jaw with her thumb; she can then release her thumb. Insome embodiments, she may perform this maneuver such that the distal endof the needle's shaft 2030 is flush with one end of the clip 2010. Thespring may force the jaws 2010 to grip the needle shaft 2030, ratherthan being secured by a screw. In some embodiments, this designaccommodates needles of varying shaft diameters. Also, in variousembodiments, the needle 2020 is inserted by moving in a directionroughly perpendicular to its shaft axis and this may create a situationin which the tip of the needle does not come close to the jaws. This mayreduce the chance of damage to the needle tip from the jaws, andfurther, allow mount to be removed even if the needle is still embeddedin tissues.

In some embodiments, the spring action may be from a metal or otherancillary spring between the plastic. In some embodiments, the springmay be a coil or bent wedge design. In some embodiments, an integratedplastic or native material spring may also be incorporated into themolded parts. In some embodiments, the clamp components may be bothmolded, machined, or a combination thereof. In some embodiments, thematerials may be medical grade plastic, which may provide light weight.The material may also be stainless steel, which may provide for easieror more economical sterilization. The apparatus can be made of anycombination of plastic, metal, and ceramic, and can be fabricated bymachining, casting, molding, or rapid prototyping (SLA, SLS, FDM, EBM,etc.). Further, in various embodiments, the needle may vibrate or heatup and the design of the needle mount's jaw accommodates this.

A second tracked device, rig, or mount 2100 of FIG. 21 may provide theframe or reference. This tracked device 2100 may be something that isdedicated for this purpose, or it may be a tracked surgical device 2100that is appropriately configured. For example, an ultrasound transducermay also be tracked. In some embodiments, the ultrasound transducer'stracking (fiducial) mount has divot, groove, etc. 2110 in it forreceiving the needle tip. This is depicted in FIG. 21. The location ofthe divot relative to the ultrasound transducer's fiducials is known.Therefore, in some embodiments, by touching the needle tip to this divotas depicted in FIG. 22 at position 2201, the system can compute theneedle tip's position relative to the fiducials attached to the needle.

Example embodiments of performing this calculation are as follows: Therigid body transformations listed below (which can be represented by 4×4matrices, quaternions, etc.) may be known by the image guidance system:

-   -   transducerFiducials_from_tracker (the position and orientation        of the ultrasound transducer's fiducials, relative to the        tracking device's reference coordinate system)    -   needleFiducials_from_tracker (the position and orientation of        the needle's fiducials, relative to the tracking device's        reference coordinate system)    -   This position is also known: (which can be presented by a        4-element vector: {x, y, z, 1}) divot_in_transducerFiducials        (the position of the divot relative to the ultrasound        transducer's fiducials)    -   In order to find needleTip_in_needleFiducials (the relative        locations or transformation between the needle tip and the        fiducials attached to the needles), various embodiments may        perform the following calculations when the needle tip touches        the divot:    -   needleTip_in_needleFiducials=divot_in_needleFiducials    -   needleTip_in_needleFiducials=divot_in_tracker*tracker_from_needleFiducials    -   needleTip_in_=divot_in_transducerFiducials*transducerFiducials_from_tracker*((needleFiducials_from_tracker)−1)

In some embodiments, the user may perform some or all of the followingactions before the calculations above are performed:

-   1) The user may hold a needle tip in the divot and simultaneously    press a foot pedal to record their positions. As an alternative to a    foot pedal, the user might ask her assistant to press a button    (because the user herself cannot press the button as she has both    hands occupied).-   2) The user may hold down a foot pedal and hold the needle in the    divot, then move the transducer and needle together for roughly one    second, while maintaining the needle in the divot, and then release    the foot pedal. This may result in more samples of positions and    relative positions and may allow the system to more accurately    determine (via regression, averaging, or other calculations) the    position of the tip of the needle, relative to the needle fiducials.

Additionally, the image guidance system may be able to detect when theneedle is in the divot by a gesture, voice command, duration in a singleposition, or any other appropriate method.

Various embodiments of these techniques may be used by any kind ofmedical professional—veterinarian, physician, surgeon, emergency medicaltechnician, physician's assistant, nurse, etc. Various embodiments usedifferent kinds of rigid needles, needle-like devices (e.g.radiofrequency or microwave ablation probe, cryo-ablation probe,cannula, optical waveguide, harmonic dissector, etc.). Variousembodiments provide for the relative locations of a scalpel (where thedivot may be replaced by a notched groove in order to locate theposition of the scalpel, its tip, etc).

Various embodiments of tracking the various devices, such as devices2100 and 2000, are discussed throughout herein and with respect toposition sensing units 310 and 340 of FIGS. 3A and 3B. If opticaltracking is used and the tracking system measures only the position ofeach fiducial, then three or more such fiducials may be affixed to theneedle in order to compute the orientation of the needle. Otherwise,fewer than three fiducials may be attached.

Rendering Techniques

Asynchronous Rendering

As noted above, real-time, live, or intraoperative data may be used andrendered in various embodiments herein. The update rate of the variousdata used may differ, and some may be slow enough, that if the entireimage were only updated at that rate, a physician may be able to noticethe update, which may be undesirable. For example, if an ultrasound wereonly updated once per second and the entire scene were only renderedonce per second, the physician is likely to notice this and find thesystem unusable. Perceivable lag can increase the risk of simulatorsickness, and the system might appear unresponsive. In variousembodiments, the 3D display herein is designed to reduce response timeand may appear to match the physician's movements with less or noperceivable lag. To accomplish this, various embodiments useasynchronous rendering. In some embodiments, no process or operation ina thread that renders the video or screen images waits for new data fromthe data from the other devices or systems, such as the tracking systemor ultrasound scanner. Instead those threads use the latest availabledata. In some embodiments, two accessory threads query the tracker (suchas position sensing units 310 and 340 in FIGS. 3A and 3B) and a videoframe grabber, which may be part of an image guidance system such asimage guidance system 300 of FIGS. 3A and 3B. These threads wait astheir respective hardware delivers the requested data, and then updatethe main thread's relevant data structures as soon as they have newdata, without causing the main thread to wait. Therefore, the main orgraphical rendering thread can continue to update the image withwhatever data is available for the tracking, imaging, and other data.

Similarly, the image guidance unit's main thread may instruct associatedgraphics hardware to swap the front and back display buffers immediatelyafter drawing a new image in the back buffer, without waiting for thevertical sync signal. This allows the newly drawn graphics to appear onthe display monitor sooner, and may allow for a higher graphics framerate. Using this asynchronous technique a user might notice tearing inboth the ultrasound image and in the graphics display. However, in someembodiments, the image guidance system grabs video frames much fasterthan an imager, such as an ultrasound scanner generates frames. Theimage guidance system may also draw frames much faster than the refreshrate of the LCD display monitor. As such, any tearing between successiveframes will be evenly distributed, and may be less noticeable to thehuman eye. At a 60 Hz video refresh rate, we would expect that 17 ms (1/60 Hz) of latency may be avoided by not waiting for vertical sync. Insome embodiments, latency may be reduced by up to 70 ms (or more).

These techniques may allow for low latency without requiring the varioussub-systems to necessarily be tuned to each other or wait for eachother.

Other Exemplary Embodiments for Various Procedures

Removal of Fibroids

In some embodiments, the image guidance system may be used to removefibroids, while leaving the uterine muscle wall strong enough to carry afetus. For example, when a physician finds a fibroid with a trackedlaparoscopic ultrasound, the image guidance system, such as system 300of FIGS. 3A and 3B, may display a visualization of a dissection toolrelative to the fibroid in the ultrasound image. This may help thephysician cut the muscle wall down the midline of the fibroid, or otherdesired cut approach. Such embodiments may reduce the damage to themuscle wall, which may then be more likely to support pregnancy. Theseembodiments may also reduce the amount of stitching, and reduce the timeto remove a fibroid.

In some embodiments, the system may be configured to allow for ablationof fibroids. For example, system 300 of FIGS. 3A and 3B may include atracked ablation antenna 345 and a tracked external ultrasound probe355, laparoscopic ultrasound transducer 355, etc. The image guidanceunit 330 may provide guidance information on the display 320, e.g.,including the relative positions or emplacements of the fibroid in theultrasound image and the tip, projection, and/or expected ablationvolume of the ablation needle 345. This guidance information may allowthe physician to more quickly locate and place the ablation needle intofibroids.

Today, surgeons often target fibroids 3-4 cm wide. In some embodiments,a surgeon may be able to find smaller fibroids (such as those 1 cm wideand smaller) because of the accuracy of the tracking and imaging,thereby increasing the probability of the patient carrying a baby toterm, and decreasing other symptoms resulting from fibroids.

Ablation Of Pancreatic Cysts

Pancreatic cysts may be a precursor to pancreatic cancer. Therefore, itmay be useful to ablate the pancreatic cysts when they occur, whether ornot it is certain that pancreatic cancer would necessarily follow.

In some embodiments, the image guidance system may be used aid aphysician in ablating the pancreatic cysts. For example, in someembodiments, an image guidance system, such as the system 300 of FIGS.3A and 3B may include an endoscopic ultrasound transducer 355 and anablator 345, such as an ablation needle 345. In some embodiments, theablator uses laser light, microwave, radio wave, or any otherappropriate ablation energy or technique. In some embodiments, theultrasound transducer is inserted via the mouth and images the pancreasthrough the wall of stomach or duodenum, while the ablation needleenters the patient from outside the body.

Hysteroscopy

Some physicians remove fibroids using a hysteroscope, or other flexibleendoscope that passes through the vagina and cervix and functions insidethe uterus. Hysteroscopy may be less invasive that other forms oflaparoscopic surgery because of the lack of incision and insufflation.Further, hysteroscopy can sometimes be performed in a clinic instead ofa hospital, thereby potentially reducing costs.

In some embodiments, a hysteroscope is tracked and imaged by an imageguidance system, such as system 300 of FIGS. 3A and 3B. In order toimage fibroids, a physician may use external ultrasound, filling thebladder with water so that she can image through it and see the uterus.In some embodiments, a resectoscope, which may be a hysteroscope with awire loop extending from it, may be used by the physician and with anembodiment of the image guidance system. The resectoscope may have asemi-circular loop in a plane parallel to the image plane that cantranslate forward and back (toward and away from the lens). The wireloop may be energized (e.g., electrocauterizer) and carve away adetected fibroid. In some embodiments, the image guidance system maytrack the resectoscope or hysteroscope and render the resector wireloop, relative to the ultrasound scan or any other devices used.

Harvesting Eggs

In some embodiments, the image guidance system is used to track andvisualize the ultrasound data as well as the needle that is used tocollect the eggs from the ovary. For example, in order to harvest eggs atransvaginal ultrasound probe to visualize the follicles in the ovary,which may contain eggs, may be used. The image guidance may help thephysician get a flexible needle (16 gauge, 30 cm long) into eachfollicle, through the vaginal wall. A physician may push on the outsideof the patient to push the ovary into a position where it can be imagedand accessed through the vaginal wall. Each follicle containing an eggis typically 1-2 cm wide. The physician may drain (aspirate) thecontents of the follicle, and then examine the fluid to look for an egg.The physician may then proceed to the next follicle. She may collect9-10 eggs, or even more. Eggs are often attached to the side of thefollicle, and the needle should enter the center of the follicle inorder to safely remove it from the wall. Embodiments herein make thattargeting easier by tracking the needle and the ultrasound (or otherimaging) that is used to find the eggs. Such embodiments used for thisprocedure may be a more effective procedure than is currently available.

Embryo Attachment

In some embodiments, the image guidance system is used for embryoattachment or embryo transfer. Embryos are inserted via a flexiblecatheter through the cervix. The catheter consists of a flexible innertube within a more rigid external tube, each about 10-20 cm long. Whilethe inner tube may be very flexible, the outer tube may be stiffer andallows a physician to guide the inner tube. The physician may fill thebladder with water, and uses external ultrasound to image the uterusthrough the bladder. The ideal place to implant the embryos is the“maximal implantation potential (MIP) point”, which is roughly the “top”of the uterus, between the fallopian tubes. A surgeon may use ultrasoundto find this point (possibly marking the point as discussed herein), andguide the catheter there. The goal is to implant between the two layersof the uterine lining, but “it's hard to see where the tip goes” once itis inside the uterine lining.

The catheter and/or the tip of the inner tube may be tracked and itsemplacement relative to the ultrasound image may be displayed to thephysician via the image guidance system. For example, the tip of thecatheter may be tracked and its real-time emplacement shown relative tothe ultrasound image or marked MIP. In some embodiments, in addition totracking the very tip of the inner catheter, the image guidance systemalso tracks one or more points along the catheter. As such, the imageguidance system can display the catheter's shape near its tip.

If a physician can get the embryo into the right place, it may increasethe overall success rate. This, in turn, could eventually allowphysicians to implant fewer embryos, perhaps reducing the “twin rate.”

The processes, computer readable medium, and systems described hereinmay be performed on various types of hardware, such as computer systemsor computing devices. In some embodiments, position sensing units 310and 340, display unit 320, image guidance unit 330, and/or any othermodule or unit of embodiments herein may each be separate computingdevices, applications, or processes or may run as part of the samecomputing devices, applications, or processes—or one of more may becombined to run as part of one application or process—and/or each or oneor more may be part of or run on a computing device. Computing devicesor computer systems may include a bus or other communication mechanismfor communicating information, and a processor coupled with the bus forprocessing information. A computer system or device may have a mainmemory, such as a random access memory or other dynamic storage device,coupled to the bus. The main memory may be used to store instructionsand temporary variables. The computer system or device may also includea read-only memory or other static storage device coupled to the bus forstoring static information and instructions. The computer systems ordevices may also be coupled to a display, such as a CRT, LCD monitor,LED array, e-paper, projector, or stereoscopic display. Input devicesmay also be coupled to the computer system or device. These inputdevices may include a mouse, a trackball, touchscreen, tablet, footpedal, or cursor direction keys. Computer systems or devices describedherein may include the image guidance unit 330, first and secondposition sensing units 310 and 340, and imaging unit 350.

Each computer system or computing device may be implemented using one ormore physical computers, processors, embedded devices, fieldprogrammable gate arrays (FPGAs), or computer systems or portionsthereof. The instructions executed by the computer system or computingdevice may also be read from a computer-readable medium. Thecomputer-readable medium may be non-transitory, such as a CD, DVD,optical or magnetic disk, laserdisc, flash memory, or any other mediumthat is readable by the computer system or device. In some embodiments,hardwired circuitry may be used in place of or in combination withsoftware instructions executed by the processor. Communication amongmodules, systems, devices, and elements may be over a direct or switchedconnections, and wired or wireless networks or connections, via directlyconnected wires, or any other appropriate communication mechanism.Transmission of information may be performed on the hardware layer usingany appropriate system, device, or protocol, including those related toor utilizing Firewire, PCI, PCI express, CardBus, USB, CAN, SCSI, IDA,RS232, RS422, RS485, 802.11, etc. The communication among modules,systems, devices, and elements may include handshaking, notifications,coordination, encapsulation, encryption, headers, such as routing orerror detecting headers, or any other appropriate communication protocolor attribute. Communication may also messages related to HTTP, HTTPS,FTP, TCP, IP, ebMS OASIS/ebXML, DICOM, DICOS, secure sockets, VPN,encrypted or unencrypted pipes, MIME, SMTP, MIME Multipart/RelatedContent-type, SQL, etc.

Any appropriate 3D graphics processing may be used for displaying orrendering, including processing based on OpenGL, Direct3D, Java 3D, etc.Whole, partial, or modified 3D graphics packages may also be used, suchpackages including 3DS Max, SolidWorks, Maya, Form Z, Cybermotion 3D,VTK, Slicer, or any others. In some embodiments, various parts of theneeded rendering may occur on traditional or specialized graphicshardware. The rendering may also occur on the general CPU, onprogrammable hardware, on a separate processor, be distributed overmultiple processors, over multiple dedicated graphics cards, or usingany other appropriate combination of hardware or technique.

As will be apparent, the features and attributes of the specificembodiments disclosed above may be combined in different ways to formadditional embodiments, all of which fall within the scope of thepresent disclosure.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orstates. Thus, such conditional language is not generally intended toimply that features, elements and/or states are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or states are included or are to beperformed in any particular embodiment.

Any process descriptions, elements, or blocks in the processes, methods,and flow diagrams described herein and/or depicted in the attachedfigures should be understood as potentially representing modules,segments, or portions of code which include one or more executableinstructions for implementing specific logical functions or steps in theprocess. Alternate implementations are included within the scope of theembodiments described herein in which elements or functions may bedeleted, executed out of order from that shown or discussed, includingsubstantially concurrently or in reverse order, depending on thefunctionality involved, as would be understood by those skilled in theart.

All of the methods and processes described above may be embodied in, andfully automated via, software code modules executed by one or moregeneral purpose computers or processors, such as those computer systemsdescribed above. The code modules may be stored in any type ofcomputer-readable medium or other computer storage device. Some or allof the methods may alternatively be embodied in specialized computerhardware.

It should be emphasized that many variations and modifications may bemade to the above-described embodiments, the elements of which are to beunderstood as being among other acceptable examples. All suchmodifications and variations are intended to be included herein withinthe scope of this disclosure and protected by the following claims.

What is claimed is:
 1. A method for image management in image-guidedmedical procedures, comprising: determining, with one or more computingdevices, pose information for a set of 3D visualizable medical data;determining, with the one or more computing devices, real-time poseinformation for a medical device; determining a region of interest forthe set of 3D visualizable medical data based on the real-time poseinformation for the medical device and the pose information for the setof 3D visualizable medical data; generating image guidance information,with the one or more computing devices, based on at least the 3Dvisualizable medical data in the region of interest; and displaying, onone or more displays, a graphical rendering of the image guidanceinformation.